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EXECUTIVE  SUMMARY 


Army  PM  JCALS,  under  the  direction  of  the  CALS  Test  Network  Office  (CTNO),  has  developed 
Computer-assisted  Data  Acceptance  (CAD  A)  procedures  for  automating  the  acceptance  of  CALS 
raster,  Type  I,  data.  The  objective  of  these  procedures  is  to  reduce  the  labor  requirements  of  the 
currendy  used  manual  quality  assurance  (QA)  procedures.  Effective  implementation  of  the  CADA 
procedures  depends  on  the  use  of  reliable  image  quality  analysis  and  identification  recognition 
techniques  and  tools.  ThiS  report  presents  the  results  of  testing  these  techniques  and  tools  on  a 
large  base  of  CALS  digital  data.  The  tests  were  conducted  at  the  Army  CTN  test  bed,  within  the 
CALS  Technology  Center  (CTC)  of  Army  PM  JCALS,  Fort  Monmouth,  NJ. 

The  initial  step  in  this  test  effort  was  to  develop  suites  of  test  data  for  both  image  and  identification 
recognition  techniques  testing.  For  image  quality  testing,  two  different  suites  of  test  data  were 
used.  One  suite  (Image  Data  Set  1)  represented  the  format  and  quality  of  CALS  contractor 
procured  data.  The  second  set  (Image  Data  Set  2)  consisted  of  a  representative  subset  from  some 
5,000  engineering  drawings  obtained  from  the  Army  and  Air  Force  DSREDS  and  EDCARS 
repository  sites.  For  identification  recognition,  three  different  sets  of  test  data  were  used.  ID 
Data  Set  I  was  sent  to  the  selected  neural  network  vendors  to  test  their  products  at  their  sites.  ID 
Data  Set  2  was  a  larger  and  different  set  of  DSREDS/EDCARS  test  data  that  was  used  to  test  the 
vendors’  recognition  engines  independently  at  the  Army  CTN  test  bed.  ID  Data  Set  3  was 
developed  to  test  the  vendors’  recognition  algorithms  on  a  suite  of  test  data  that  closely  represented 
good  quality  data  expected  to  be  procured  from  CALS  contractors  in  the  future. 

A  number  of  image  quality  analysis  techniques  were  tested,  including  run  length  ratio,  orphan 
analysis,  approximate  black  and  white  orphan  ratios,  peak  tile  noise,  fill  factor/compression  ratio 
ranges,  border  clipping,  and  verticality  analysis.  Extensive  testing  on  the  two  suites  of  test  data 
resulted  in  two  different  combinations  of  techniques  and  parameters.  One  combination  established 
“high  quality”  criterion  for  the  data  expected  to  be  received  when  procuring  CALS  data.  The 
second  combination  was  for  “low  quality”  criterion  for  accepting  data  that  exists  today  and  has 
been  developed  over  the  past  30  to  40  years.  Applying  the  high  quality  criterion  to  Image  Data  Set 
1  resulted  in  a  False  Accept  (FA)  rate  of  0.68  %  and  a  False  Reject  (FR)  rate  of  6. 1 8%.  (The  goal 
is  to  accept  less  than  1%  of  the  bad  images  in  a  batch  and  reject  less  than  25%  of  the  good  images 
in  a  batch.  All  rejected  images  would  be  visually  inspected  by  an  operator,  so  all  FR  would  be 
accepted.)  Applying  the  high  quality  criterion  to  Data  Set  2,  resulted  in  a  FA  of  0  %  and  a  FR  of 
good  images  of  76%.  The  FR  is  obviously  too  high.  Applying  the  low  quality  criterion  to  the 
same  set  of  images  resulted  in  a  FA  of  0. 1 8%  and  a  FR  of  10.53%  which  is  reasonable. 

Testing  of  the  identification  test  data,  ID  Data  Set  1,  by  the  vendors  quickly  revealed  the  lack  of 
preprocessing  capabilities  for  the  variety  of  title  block  and  quality  of  data  provided  in  ID  Data  Set 
1.  Four  vendors  were  selected  to  provide  their  products  for  further  testing  within  the  Army  CTN 
test  bed.  Each  of  these  four  vendors’  products  were  tested  with  ID  Data  Set  2.  The  average 
percentage  of  correctly  recognized  fields  and  characters  was  found  to  be  28  and  48  per  cent 
respectively.  The  low  percentage  of  field  recognition  indicates  the  inability  to  segment  and  remove 
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Unes  during  the  preprocessing  stage.  The  low  percentage  of  character  recognition  indicates  the 
vendors’  tools  weatoess  in  recognizing  poor  quality  hand  printed  characters  and/or  touching 
characters.  The  obvious  conclusion  was  that,  for  this  t3q)e  data,  the  preprocessing  and  recognition 
capabilities  of  these  products  would  not  be  acceptable.  The  same  vendor  products  were  tested  with 
ID  Data  Set  3.  In  this  case,  the  average  correctly  recognized  fields  and  characters  were  61  %  and 
74%  respectively.  One  of  the  vendors  was  in  the  90%  range.  It  is  expected  diat  with  proper 
preprocessing/postprocessing  of  CALS  qucility  data  and  training  on  both  alpha  and  numeric 
characters  the  correct  recognition  will  be  in  the  upper  90%  range. 

The  recommendations  in  Section  6  strongly  suggest  that  procurement  contracts  specify  the  ANSI 
Y14.1  and  YI4.2M  standards  be  followed  closely,  in  order  to  implement  automated  CAD  A 
procedures  for  the  acceptance  of  CALS  digital  data. 

The  next  steps  are  to  select  the  vendors  products  that  will  be  recommended  for  licensing  by  the 
government,  integrate  these  techniques  and  processing  tools,  test  them  on  government  provided 
CALS  raster  Type  I  data,  and  deliver  them  to  the  Services  for  field  operational  testing  and 
evaluation. 
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DISCLAIMER 
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SECTION  1 

INTRODUCTION 


1.1  BACKGROUND 

Data  is  so  important  to  the  logistic  support  of  the  Department  of  Defense’s  (DoD)  weapon  systems 
that  every  effort  must  be  made  to  insure  that  the  integrity  of  the  data  is  maintained,  from  its 
procurement  through  its  ultimate  use  in  the  field.  The  procurement  contract  must  clearly  specify  in 
the  supporting  documents  (Contract  Data  Requirements  Lists  [CDRLs],  Data  Item  Descriptions 
IDIDs],  Statements  of  Work  fSOWs],  etc.)  the  CALS  Standards  and  Specifications,  that  the 
contractor  is  responsible  for  producing,  testing,  and  delivering  quality  data  and  must  also  define 
the  government’s  data  acceptance  requirements  at  the  contractor’s  site,  as  well  as  at  the  user 
repository  site.  The  contract  may  include  penalty  and  incentive  clauses  relative  to  receiving  quality 
data,  and  should  address  the  certification  and  warranty  of  the  data  in  order  to  maintain  data  integrity 
in  the  future. 

The  data  management  systems  fielded  at  the  Army  DSREDS  sites,  the  Air  Force  EDCARS  sites, 
and  the  Navy  EDMICS  sites  all  have  the  resources  to  accept  digital  data.  Manual  procedures  exist 
that  can  be  implemented  at  these  sites,  yet  field  testing  proved  that  other  means  should  be  employed 
to  reduce  the  labor  intensive  effort  of  viewing  each  image  at  one  or  more  workstations.  Providing 
100%  quality  assurance  of  each  image  by  qualified  engineering  data  personnel  at  the  repository 
may  seem  to  be  the  best  means  of  data  acceptance,  but  is  costly,  and  perhaps,  impractical  for  the 
loading  of  huge  amounts  of  data. 

The  DoD  recognized  that  the  problems  of  data  quality,  and  the  distribution  of  the  data,  could  best 
be  solved  by  developing  standards  for  the  development  and  distribution  of  data  in  an  electronic 
format.  The  Computer-Aided  Acquisition  and  Logistic  Support  (CALS)  initiative,  introduced  in 
1985,  provided  the  basis  for  obtaining  quality  product  and  technical  publication  data,  in  standard 
formats  on  standard  media.  Industry  has  been  cooperating  with  the  DoD  in  the  development  and 
testing  of  these  standards.  As  part  of  the  infrastructure  modernization  effort,  the  CALS  policy 
office  directed  the  CALS  Test  Network  (CTN)  to  develop  procedures  for  the  acceptance  of  product 
and  technical  publication  data  in  CALS-compliant  format.  The  task  was  assigned  to  Army  PM 
JCALS. 

The  Computer-Assisted  Data  Acceptance  (CADA)  procedures  developed  at  the  Army  CTN  CALS 
test  bed,  under  the  auspices  of  the  CALS  Technology  Center  (CTC),  for  the  CALS  Test  Network 
Office  (CTNO),  focus  on  the  acceptance  of  production  (Level  III)  engineering  drawing  data.  It  is 
assumed  that  the  technical  content  accuracy  has  been  verified  prior  to  data  acceptance  and  is  an 
issue  during  data  acceptance  only  if  the  digitized  drawing's  readability  and  reproducibility  is 
impaired.  The  quality  of  the  accepted  digital  data  must  be  maintained  during  the  process  of 
conversion,  storage,  and  distribution  by  the  tri-service  repositories. 


1 


Test  Report:  Phase  III  Computer-Assisted  Data  Acceptance 

The  use  of  computer-assisted  techniques  poses  certain  risks.  No  technique  or  combination  of 
techniques  will  detect  where  a  single  dimension,  revision  letter,  or  description  was  hand  drawn  by 
using  a  hard  leaded  pencil  (not  conforming  to  MIL-STDs),  and  upon  being  scanned,  resulted  in 
missing  information.  Of  course,  these  risks  exist  today  and  qualified  inspectors  do  not  always 
detect  these  single  instance  cases.  If  the  techniques  are  not  thoroughly  tested,  and  the  results  are 
the  false  acceptance  of  poor  quality  data,  which  can  then  be  transposed  to  the  repositories*  mass 
optical  disk  storage  media,  then  this  too  is  risky.  If  the  implementation  of  the  techniques  requires 
excessive  repository  computer  power,  or  has  an  impact  on  the  basic  mission  of  the  repository,  then 
this  is  also  unwarranted.  Not  only  must  the  CADA  techniques  be  used  with  confidence;  but,  how 
they  are  used,  by  whom  they  are  used,  and  where  they  are  used,  must  be  considered  with  care. 

The  rights  to  use  the  techniques  is  another  issue  that  must  be  faced.  When  the  integrated 
techniques  have  been  tested  and  recommendations  made,  then  the  government  must  obtain  license 
rights  to  those  techniques  that  are  proprietary.  The  government  must  have  full  use  of  the 
techniques  to  implement  the  CADA  procedures  in  the  field. 

The  automation  of  the  identification  data  verification  involves  the  application  of  techniques  to 
"recognize,"  within  the  image  area,  the  key  identifiers  such  as  drawing  numbers,  drawing  sizes, 
drawing  revisions,  etc.  The  "recognized"  ASCII  data  will  then  be  compared  with  the  CALS  key 
ID  header  data  "keyed  in"  by  an  operator. 

The  usefulness  of  CADA  procedures  are  highly  dependent  on  the  techniques  and  tools  used  to 
implement  them.  Practical  use  of  the  procedures  by  qualified  personnel  dictates  that  they  can  be 
used,  with  confidence,  to  reject  poor  quality  image  and  identification  data.  It  is,  therefore, 
essential  that  the  techniques  be  fully  tested  in  both  the  laboratory  and  the  field  prior  to  formally  ! 
releasing  them.  This  report  addresses  the  testing  within  the  laboratory  environment  at  the  Army 
CTN  test  bed. 

1.2  PURPOSE 

The  purpose  of  this  test  phase  is,  first,  to  thoroughly  test  image  and  identification  data  (ID) 
algorithms  and  tools  against  quality  data  that  meet  the  CALS  standards. 

A  second  purpose  is  to  evaluate  the  use  of  these  algorithms  and  tools,  or  others,  to  accept  existing 
data  obtained  from  the  Air  Force  and  Army  repositories. 

1.3  SCOPE 

The  testing  involves  obtaining  “live”  data  from  DSREDS  and  EDCARS  sites,  converting  the  data 
to  CALS  format  and  then  classifying  the  data  for  testing. 

Both  Image  and  ID  testing  of  the  converted  “live”  data  was  performed,  and  the  results  analyzed 
and  presented  in  a  test  report.  Image  and  ID  algorithms  were  used  to  compare  the  acceptance  of 
contractor  type  quality  data,  with  the  acceptance  of  representative  existing  data. 
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1.4  APPROACH 

The  tests  defined  herein  concentrate  on  testing  image  and  ID  techniques  for  automating  the 
acceptance  of  quality  product  data  in  CALS  raster  type  I  format  that  is  expected  to  be  received  from 
a  contractor  and  also  to  utilize  the  same  techniques,  and  others,  to  test  the  acceptance  of  existing 
engineering  drawing  data  obtained  from  the  Army  and  Air  Force  repositories  (DSREDS  and 
EDGARS  respectively). 

The  approach  taken  in  this  test  effort  for  both  the  image  and  ID  techniques  evaluations  were: 

1 .  develop  a  test  plan  (see  Appendix  A); 

2.  identify  non-proprietary  advanced  techniques  to  be  used  and  develop  new  ones 
where  necessary  for  image  analysis; 

3.  select  neural  network  technology  for  conducting  “pre-processing”  and 
“recognition”  of  the  key  ID  within  a  suite  of  engineering  drawing  test  data; 

4.  develop  and  classify  suites  of  “quality”  image  and  ID  test  data; 

5.  obtain  existing  DSREDS/EDCARS  data,  translate  them  into  CALS  format,  analyze 
and  classify  them  to  build  a  suite  of  image  and  ID  test  data; 

6.  conduct  image  testing  of  existing  and  representative  “quality”  test  suites  using 
different  combinations  of  image  algorithms  and  techniques; 

7.  conduct  evaluation  of  selected  vendor  neural  network  techniques  via  analysis  of  the 
suite  of  existing  DSREDS/EDCARS  test  data  and  the  suite  of  “representative” 
contractor  derived  quality  data; 

8.  document  the  test  results  and  analyze  them;  and 

9.  present  the  results  in  a  test  report. 

The  image  techniques  and  the  neural  network  vendors’  products  are  described  within  the  body  of 
this  report.  The  test  results  are  summarized  in  tabular  format  and  the  detailed  test  results  are 
contained  in  the  appendices.  The  test  data  were  prepared  from  a  cross-section  of  the  data  obtained 
from  DSREDS/EDCARS  sites  and  tu-e  representative  of  both  poor  and  good  quality  existing  data. 

It  should  be  noted  that  some  of  the  “quality  ID  test  data”  was  produced  by  replacing  broken  lines 
and  removing  excessive  noise,  since  this  type  of  data  is  not  expected  to  be  received  from  the 
contractor  when  procuring  CALS  data. 
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SECTION  2 

TEST  DATA  REQUIREMENTS  FOR  CADA  TECHNIQUES 


2.1  IMAGE  TEST  DATA  PREPARATION  AND  CLASSIFICATION 

The  test  data  for  image  quality  analysis  is  of  two  distinct  categories.  The  first  category  was  used 
for  “high  quality”  testing  (i.e.,  treating  the  data  as  if  it  was  delivered  by  a  contractor  and  applying 
stringent  quality  requirements  to  it).  The  second  category  was  of  “live”  data  (i.e.,  existing  data 
fi'om  government  repositories  like  DSREDS  and  EDGARS).  The  quality  requirements  for  this 
category  were  much  lower  as  compared  to  the  first  one.  The  following  two  subsections  describe 
the  preparation  and  classification  effort  for  the  two  categories  of  data. 

2.1.1  Image  Data  Set  1 

The  tri-services  provided  the  source  data  that  was  used  to  prepare  this  suite  of  test  data.  Aperture 
cards  from  the  Air  Force  and  the  Navy  were  received  and  a  few  were  selected  for  use  in  building 
the  suite  of  test  data.  The  CECOM  DSREDS  site  provided  original  hard  copies,  which  were 
analyzed,  and  a  number  of  high  quality  vellum  and  mylar  drawings  were  selected  that  contained  a 
representative  sample  of  sparse  and  dense  drawings  that  ranged  from  “A”  size  through  “E”  size. 
The  CECOM  engineering  data  personnel  utilized  their  camera  and  aperture  card  resources  to 
produce  a  suite  of  aperture  card  test  data.  The  aperture  cards  included  both  good  and  bad  data. 

The  bad  data  cards  contained  overexposed  and  underexposed  images  that  ranged  from  very  dark  to 
very  light.  The  result  was  a  data  base  of  some  300  aperture  cards  of  which  about  one-third  were  of 
good  quality  and  about  two-thirds  were  of  poor  quality. 

These  aperture  cards  were  then  sent  to  a  vendor,  who  scanned  them,  without  enhancements,  to 
produce  digitized  data,  on  magnetic  tape,  with  the  images  compressed  in  TIFF  CCITT  Group  IV 
format  The  punched  hollerith  data,  from  each  aperture  card,  was  stored  in  the  TIFF  header  of  the 
image  file. 

After  the  files  were  received  from  the  vendor,  they  were  tested  for  proper  CCITT  Group  IV  format 
of  the  compressed  image  data.  Of  the  300  TIFF  files,  seven  had  bit  errors  in  the  compressed 
image  data  and  could  not  be  used  for  testing.  The  293  good  files  were  converted  to  CALS  format 
for  testing  purposes,  using  utility  software  developed  for  this  purpose.  For  each  file,  the  software 
obtained  information  required  for  the  CALS  header,  including  the  punched  hollerith  data,  from  the 
TIFF  header,  and  used  it  to  create  the  CALS  header  for  that  file.  The  compressed  image  data  in  the 
TIFF  file  was  already  in  the  CCITT  Group  IV  format  and  needed  no  further  conversion. 

The  test  data  was  stored  on  a  Sun  file  server  on  an  Ethernet  Local  Area  Network  (LAN).  This 
configuration  provided  a  great  deal  of  flexibility  in  the  test  center  since  the  files  were  accessible  on 
the  Sun  workstation,  and  via  the  LAN,  on  various  PC  compatible,  and  Macintosh,  workstations. 
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One  PC  compatible  image  workstation  was  used  to  store  the  entire  suite  of  data  in  CALS  format 
locally  for  viewing  purposes.  Another  workstation,  a  Macintosh,  was  used  to  view  and  make  hard 
copies  of  the  test  images. 

All  of  these  images  were  visually  examined  and  classified  as  acceptable  or  unacceptable.  Since  the 
assumption  was  that  the  data  obtained  from  the  contractors  should  be  of  very  high  quality,  these 
images  were  very  critically  examined  and  accepted  only  if  they  were  extremely  clear.  Out  of  the 
293  images,  only  82  were  considered  acceptable  on  visual  inspection. 

2.1.2  Image  Data  Set  2 

To  examine  the  performance  of  CADA  techniques  on  “live”  data,  5,194  engineering  drawings 
were  obtained  from  five  different  DSREDS/EDCARS  sites.  The  data  was  received  via  9-track 
tapes  in  DSREDS/EDCARS  format,  which  was  different  from  the  CALS  format.  Therefore,  a 
conversion  from  DSREDS/EDCARS  format  to  CALS  format  was  recjuired.  An  Intergraph 
machine,  obtained  from  MI  COM,  was  used  for  this  specific  purpose,  since  the  utilities  for 
DSREDS  to  CALS  conversion  were  available  on  that  machine.  Each  tape  was  first  loaded  into  the 
Intergraph  and  conversion  utilities  were  run.  The  converted  files  were  then  loaded  onto  a  Sun  4 
machine,  where  they  were  stored  on  a  1.3  GB  hard  drive.  The  Sun  4  was  on  a  local  area  network 
allowing  the  files  to  be  accessed  from  different  machines,  such  as  EBM  PCs  and  Macintoshes. 

These  images  were  then  visually  examined,  and  a  subset  of  541  images  that  provided  a  good 
representation  of  the  entire  set,  were  selected.  This  subset  was  further  examined  to  classify  the 
images  as  either  acceptable  or  non-acceptable.  The  criteria  used  for  acceptance  was  fairly  relaxed. 
As  long  as  the  important  areas  of  a  drawing  were  legible,  it  was  considered  acceptable,  even  if  it 
contained  noise  and/or  faded  areas.  Out  of  the  541  images,  527  were  deemed  acceptable  and  only 
14  unacceptable  upon  visual  inspection. 

2.2  ID  TEST  DATA  PREPARATION  AND  CLASSIFICATION 

Three  sets  of  test  data  were  prepared  for  performing  identification  recognition  analysis  and  testing. 
These  data  sets  correspond  to  the  three  segments  of  identification  recognition  analysis  and  testing: 

1  training  and  prelimintuy  testing  by  the  candidate  identification  recognition  vendors; 

2  testing  of  candidate  vendor  products  on  data  that  is  of  the  quality  found  in 
government  repositories  today;  and 

3  testing  of  candidate  vendor  products  on  data  that  meet  CALS  standards,  and  of  the 
quality  expected  to  be  delivered  by  government  contractors  in  the  future. 

The  elements  of  these  data  sets  were  prepared  by  extracting  the  title  block  areas  from  raster  images 
of  engineering  drawings,  with  the  use  of  a  raster  editor.  These  sets  are  representative  samples  of 
the  various  sizes,  vendors,  and  styles  of  title  blocks.  Figure  2-1  shows  a  typical  title  block.  These 
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title  block  images  were  saved  in  TIFF  uncompressed  format,  prior  to  their  use  in  the  identification 
recognition  analysis  and  testing  process.  The  details  of  these  test  sets  are  presented  in  the 
paragraphs  that  follow. 

2.2.1  ID  Data  Set! 

Data  Set  1  was  prepared  by  classifying  data  types  and  sizes  for  use  by  the  candidate  identification 
recognition  vendors.  The  candidate  vendors  were  to  use  this  set  to  train  their  products  and  to 
conduct  self-tests.  The  purpose  of  this  test  set  was  to  show  the  candidate  vendors  the  type  of  data 
that  was  to  be  processed  by  their  product.  Examples  from  this  set  are  shown  in  Figure  2-2.  As  the 
figure  shows,  the  quality  ranged  from  very  clear  to  barely  legible. 

2.2.2  ID  Data  Set  2 

Data  Set  2  was  prepared  by  the  Army  CTN  test  bed  personnel  for  testing  of  candidate  vendor 
products  in  the  test  bed.  This  test  set  was  extracted  from  engineering  data  supplied  by 
DSREDS/EDCARS  and  is  considered  typical  of  the  data  currently  stored  in  the  government’s 
repositories.  Examples  of  this  data  are  shown  in  Figures  2-3.  The  examples  show  a  range  of 
quality  that  is  present  in  current  repository  data  along  with  the  non-standard  nature  of  their  layouts. 

2.2.3  ID  Data  Set  3 

Data  Set  3  was  prepared  by  the  Army  CTN  test  bed  personnel  for  testing  of  candidate  vendor 
products  in  the  test  bed.  This  test  set  was  also  extracted  from  engineering  data  supplied  by 
DSREDS/EDCARS.  It  was  manually  enhanced  to  reflect  CALS  type,  higher  quality  data  that  is 
expected  to  be  delivered  by  the  government  contractors  in  the  future.  Enhancements  included 
broken  line  repair,  speckle  removal,  and  character  separation.  Examples  of  this  data  are  shown  in 
Figure  2-4.  The  examples  show  some  of  the  enhancements  that  were  made  which  simulate  the 
higher  quality  data  that  is  expected  to  be  delivered  in  the  future. 

2.3  DATA  QUALITY 

The  quality  of  the  CALS  digital  data  delivered  to  the  government  dictates  the  amount  of  quality 
control  (QC)  inspection  required  and  directly  impacts  the  cost  of  accepting  the  data  whether  by 
manual  procedures,  visual  quality  assurance  (QA),  or  the  use  of  CADA  procedures.  Every  effort 
must  be  made  to  insure  that  data  quality  is  emphasized  at  every  stage,  from  procurement  to  storage 
of  the  data  at  the  receiving  repository. 

Data  quality  means  different  things  to  different  people.  Data  quality  will  be  discussed  herein  only 
as  it  relates  to  the  use  of  the  techniques  and  tools  to  automate  the  acceptance  of  quality  digital  data. 
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Figure  2-2.  Examples  from  Test  Set  1 
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2.3.1  Definitions 

Quality  is  defined  in  many  ways.  Most  definitions  have  been  directed  towards  hard  copy  or 
microform  documents.  MIL-M-9868D  is  the  standard  for  the  acceptance  of  microfilm  engineering 
documents,  and  does  define  some  qualities  and  defects  with  regard  to  microfilm  image 
representations.  Some  of  these  definitions  can  be  adapted  to  the  application  of  quality  as  it  relates 
to  raster  graphics  representations;  however,  no  clear  definition  of  quality  for  raster  images  is 
known  to  exist. 

For  purposes  of  DoD  engineering  drawings  acceptance,  quality  implies  the  clarity  and  fidelity  of 
the  digital  representation.  The  clarity  and  fidelity  must  be  such  that,  when  retrieved  from,  or 
reproduced  at,  the  repository,  the  end  user  can  read,  interpret,  and  measure  objects  depicted  in  the 
drawing  image  area,  and  clearly  interpret  the  key  ID  data  within  the  title  block  or  tabular  fields  of 
the  drawing.  In  terms  of  computer-assisted  data  acceptance,  the  quality  must  be  measured 
objectively  in  such  things  as  line  widths,  run  lengths,  compression,  resolution,  etc. 

2.3.2  Data  Quality  Issues 

The  CALS  MIL-STD-1840A  and  MIL-R-28002A  define  the  requirements  for  the  production  and 
delivery  of  product  data  in  digital  format.  The  format  and  validity  of  the  deliverable  data  files,  the 
raster  image  quality  and  the  quality  of  the  deliverable  file  header  ID  are  all  quality  issues  that 
should  be  considered. 

Image  quality  is  an  issue  primarily  when  evaluating  raster  data  that  was  scanned  and  digitized  from 
hard  copy,  film,  or  microfilm  sources.  Tlie  legibility  of  the  raster  image  may  be  poor  because  the 
source  image  legibility  was  poor  or  because  noise  was  introduced  by  the  digitizing  process.  The 
image  may  be  skewed  because  the  source  image  on  microfilm  or  hard  copy  was  skewed,  or 
because  the  source  image  was  skewed  when  it  was  scanned  and  digitized.  Image  quality  is  not  as 
important  an  issue  when  dealing  with  raster  image  data  files  that  were  generated  directly  from  a 
CAD  system. 

A  related  issue  is  the  degree  of  compressibility  of  raster  images.  Dark  images  and  noisy  images 
both  have  an  excess  number  of  dark  pixels  which  are  stored  as  information  by  the  CCi'lT  Group 
rV  compression.  This  reduces  their  compressibility  and  increases  the  storage  requirements. 

The  acceptance  of  engineering  drawing  data  on  aperture  card  media  has  been  in  relation  to  their 
“legibility  and  reproducibility.”  This  is  very  subjective,  not  to  mention  that  the  QA  can  vary  from 
site  to  site.  The  use  of  computer-assisted  data  acceptance  techniques  and  tools  require  more 
objective  guidelines  in  the  preparation  and  QA  of  CALS  data.  This  issue  must  be  addressed  clearly 
in  the  data  procurement  contract,  as  well  as  in  the  automated  acceptance  of  the  data. 
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SECTION  3 

IMAGE  QUALITY  ANALYSIS  AND  TESTING 


3.1  TEST  PROCEDURES 

The  first  step  was  to  prepare  and  classify  the  data,  which  has  been  described  in  detail  in  the 
previous  section.  Then  a  set  of  iterative  steps  were  followed  as  described  below: 

Step  1:  Apply  a  set  of  techniques  (described  later)  on  the  set  of  images  being  tested 

and  tabulate  the  numerical  indexes  corresponding  to  different  techniques. 

Step  2:  Determine  the  acceptable  ranges  for  different  indexes  based  on  the  results  of 

visual  inspection.  Also  determine  a  combination  of  different  indexes  that 
gives  the  best  results. 

Step  3:  Based  on  the  ranges  established  in  Step  2,  determine  which  images  were 

falsely  accepted  or  falsely  rejected.  Visually  examine  those  images  and 
identify  peculiar  features  of  those  images  that  may  be  causing  the  wrong 
decisions.  Hypothesize  a  different  technique  or  a  different  combination  of 
existing  techniques  that  could  lower  the  occurrence  of  false  accepts/rejects. 

Step  4:  Go  back  to  Step  1  and  repeat  all  the  steps  till  satisfactory  results  are 

obtained. 

3.2  QUALITY  ANALYSIS  TECHNIQUES 

Individual  and  combined  analysis  of  a  number  of  raster  techniques  was  performed.  The  following 
is  a  list  of  all  the  main  techniques  used  in  the  test: 

•  orphan  analysis, 

•  run  length  analysis, 

•  fill  factor/Compression  ratio  ranges, 

•  peak  tile  noise, 

•  approximate  orphan  analysis, 

•  border  clipping,  and 

•  verticality  analysis. 

The  images  were  processed  using  evaluation  software  which  was  developed  to  apply  the  above 
techniques  to  CALS  format  raster  image  files.  No  proprietaiy  software  packages  were  used  in  this 
test  The  image  evaluation  software  was  based  on  the  CCITT  Group  IV  decompression  software 
developed  by  the  Lawrence  Livermore  National  Laboratory  for  the  CTN.  Code  was  inserted  into 
the  decompression  routines  to  measure  quantitative  information  about  the  image  as  it  was  being 
decompressed.  This  was  used  to  collect  statistics  about  each  image  and  to  calculate  indexes  which 
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described  the  image  quality.  These  quantitative  results  were  analyzed  with  respect  to  the  visually 
classified  quality  indexes.  For  reading  a  magnetic  tape  in  MIL-STD-1840  format,  a  tool  provided 
by  CTN  was  used.  The  evaluation  software  was  developed  in  ‘C’  language  and  the  user  interface 
was  based  on  OpenWindows. 

The  following  paragraphs  discuss  each  of  the  techniques. 

3.2.1  Orphan  Analysis 

Orphan  analysis  is  a  means  of  detecting  noise  data  in  uncompressed  image  data.  It  is  the  analysis 
of  the  frequency  of  orphan  pixels  which  are  found  in  an  image.  The  basic  premise  is  that  pixels  of 
one  color  (orphans),  which  are  completely  surrounded  by  the  opposite  color,  represent  noise  data 
instead  of  image  data. 

The  orphan  parts  per  million  (ppm)  is  used  to  determine  the  amount  of  noise  in  an  image.  Two 
indexes  are  used:  black  orphan  ppm  and  white  orphan  ppm.  Black  orphan  ppm  is  the  ratio  of 
black  orphan  pixels  to  the  total  white  area  in  the  image.  White  orphan  ppm  is  the  ratio  of  white 
orphan  pixels  to  the  total  black  area  in  the  image.  High  oiphan  ppm  indicates  a  noisy  image;  while 
low  orphan  ppm  indicates  the  opposite;  however,  it  cannot  by  itself  indicate  a  good  quality  image. 
Light  images  often  show  low  orphan  ppm  resulting  from  missing  image  areas  which  do  not  contain 
orphan  pixels. 

3.2.2  Run  Length  Analysis 

Run  length  analysis  is  a  means  of  detecting  noise  data  in  compressed  image  data.  It  is  the  analysis 
of  the  distribution  of  pixel  run  lengths  which  have  been  encoded  within  a  compressed  image.  The 
basic  premise  here,  is  that  if  the  pixel  ran  Shorter  than  a  minimum  pixel  width,  it  would  represent 
noise  rather  than  information.  This  was  derived  from  the  assumption  that  a  normal  drawing  had 
line  widths  of  .016  inch  or  greater  as  specified  in  the  ANSI  Y14.2M-1979  standard  for  engineering 
drawing  line  conventions  and  lettering.  The  ratio  of  the  minimum  line  width  to  a  single  pixel  width 
gave  the  minimum  pixel  width  of  that  line.  In  an  image  with  a  density  of  200  pixels  per  inch  (.005 
inch/pixel),  the  lines  would  be  at  least  three  pixels  wide.  Consequently,  a  run  length  histogram  of 
a  normal  image  would  have  a  distinct  peak  at  or  slightly  above  a  run  length  of  three  pixels  with  a 
sharp  cutoff  of  shorter  run  lengths.  A  noisy  image  would  have  a  peak  at  a  lower  run  length  or  no 
peak  at  all.  The  test  software  computed  the  number  of  run  lengths  in  the  one-pixel  to  10-pixel 
range  and  then  determined  the  largest  number  of  run  lengths  in  the  four-pixel  to  10-pixel  range. 

The  test  results  showed  that,  in  this  suite  of  test  data,  good  drawings  revealed  a  range  of  peak  run 
lengths  from  four  to  six  pixels,  but  predominantly  had  a  peak  run  length  of  five  pixels. 

The  run  length  ratio  was  the  index  used  to  determine  the  noise  in  an  image.  It  was  the  ratio  of  the 
number  of  one-pixel  run  lengths  to  the  number  of  peak  run  lengths.  A  high  ratio  indicated  a  noisy 
image;  while  a  low  ratio  indicated  a  noise-free  image.  The  following  ratios  were  also  tried  and  the 
results  obtained  were  similar  to  tlie  above  mentioned  ratio:  ratio  of  number  of  one-pixel  and 
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two-pixel  run  lengths  to  the  number  of  peak  run  lengths,  ratio  of  one-pixel  and  two-pixel  run 
lengths  to  the  number  of  peak,  peak-minus-one  and  peak-plus-one  run  lengths. 

3.2.3  Fill  Factor/Compression  Ratio  ranges 

This  technique  analyzed  the  fill  factor  (FF)  and  the  compression  ratio  (CR)  of  a  compressed  image. 
The  basic  premise  was  that  acceptable  images  fell  into  a  predictable  range  of  fill  factors  and 
compression  ratios.  This  was  derived  from  the  assumption  that  a  sparse  image  (low  fill  factor) 
would  be  highly  compressed  (high  compression  ratio)  while  a  busy  image  (high  fill  factor)  would 
be  compressed  to  a  lesser  degree  (low  compression  ratio). 

3.2.4  Peak  Tile  Noise 

This  technique  was  used  to  detect  images  that  were  bad  only  in  small  areas,  such  as  having  small 
faded  portions.  Noise  indexes  based  on  the  entire  image  may  fail  to  catch  such  defects.  In  this 
technique,  the  images  were  divided  into  256  pixel  X  256  pixel  tiles  and  the  noise  level  (i.e., 
number  of  run  lengths  of  one)  was  computed  for  each  tile.  An  image  was  considered  good  if  the 
maximum  noise  level  for  any  tile  was  below  an  established  threshold.  This  technique  performed 
well  in  detecting  images  that  were  faded  in  a  small  area. 

3.2.5  Approximate  Orphan  Ratio 

One  disadvantage  of  the  orphan  analysis  technique  was  that  it  runs  quite  slowly.  The  approximate 
orphan  ratio  was  yet  another  ratio,  that  gave  very  similar  results,  and  could  be  computed  much 
faster  than  an  orphan  ratio.  In  a  CCITT  Group  IV  compressed  image  file,  pbcels  in  each  scan  line 
are  coded  relative  to  the  pixels  in  the  previous  scan  line,  using  one  of  the  three  coding  modes: 
horizontal,  vertical,  or  pass  mode  (see  CCITT  Blue  Book).  Each  coding  mode  also  has  an 
associated  run  length.  A  run  length  of  one  in  either  horizontal  or  pass  mode  is  a  good  indication  of 
noise,  and  gives  information  similar  to  the  information  given  by  “orphans.” 

The  ratio  of  run  lengths  of  one  in  non-vertical  mode  to  the  total  number  of  pixels  of  the  opposite 
color  (approximate  orphan  ratio),  was  found  to  work  very  well  and  was  successfully  used  in  place 
of  exact  orphan  ratio.  Again,  the  main  advantage  of  using  this  technique  was  its  speed  as 
compared  to  orphan  analysis.  For  detecting  white  noise,  a  slightly  different  ratio,  that  of  the 
number  of  run  lengths  of  one  of  white  (in  any  mode)  to  the  total  number  of  black  pixels,  was 
found  to  provide  a  very  close  approximation  to  white  orphan  ratio. 

3.2.6  Border  Clipping 

While  testing  DSREDS/EDCARS  data,  it  was  observed  that  many  images  had  a  lot  of  noise  along 
the  borders,  causing  them  to  be  rejected.  So,  it  was  decided  to  ignore  300  pixels  (i.e.,  about  an 
inch  and  a  half)  along  the  four  borders  of  the  image  while  computing  various  noise  indexes.  This 
approach  reduced  the  number  of  false  rejects  considerably  in  the  case  of  DSREDS/EDCARS  data. 
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3.2.7  Verticality  Analysis 

Verticality  analysis  was  the  comparison  of  the  number  of  vertical  modes  in  a  compressed  image  to 
the  number  of  both  horizontal  and  pass  modes.  Since,  in  a  good  image,  consecutive  scan  lines 
would  differ  only  slightly,  the  number  of  vertical  modes  should,  therefore,  be  high  as  compared  to 
horizontal  or  pass  modes.  While  this  premise  was  found  to  be  true  in  general,  no  significant 
patterns  could  be  discovered  that  could  be  used  to  accept/reject  images  with  a  good  confidence 
level. 

3.3  TEST  RESULTS 
3.3.1  Image  Data  Set  1  Results 

As  discussed  in  section  2.1.1,  a  set  of  293  images  were  used  for  “high  quality  testing”  (i.e., 
testing  for  quality  standards  that  should  be  applicable  to  new  CALS  data  obtained  from 
contractors).  For  these  images,  any  presence  of  noise  above  a  certain  level  was  considered  to  be 
unacceptable,  even  if  the  image  was  legible  (because  among  other  things,  noise  also  causes  the 
compressed  image  to  be  larger  than  necessary,  leading  to  higher  storage  requirements).  Several 
different  techniques,  as  described  in  the  previous  subsection,  were  tried.  As  found  in  the  previous 
test  phase,  no  single  technique  was  considered  good  enough  for  obtaining  desirable  results. 
Therefore,  a  combination  of  different  techniques  was  necessary.  The  following  is  the  list  of 
techniques  that  were  used  in  the  final  combination. 

1 .  Run  length  ratio:  This  technique  was  found  to  be  very  helpful  in  detecting  noise 
in  images.  The  acceptable  limit  for  this  ratio  was  set  at  14  in  this  combination. 

2.  Approximate  black  orphan  ratio:  This  technique  was  used  as  a  substitute  for 
black  orphan  ratio  for  efficiency  considerations.  It  proved  very  helpful  in  detecting 
black  speckles,  also  referred  to  as  “pepper”.  A  cutoff  value  of  0.02  was  used. 

3.  Peak  tile  noise:  Black  run  lengths  of  1  were  counted  in  256  X  256  tiles  and  if  any 
tile  had  a  count  of  more  than  156,  the  image  was  rejected.  This  technique  proved 
helpful  in  detecting  defects  that  were  localized  to  small  areas  in  an  image  (i.e., 
fading  along  an  edge.) 

4.  Approximate  white  orphan  ratio:  Tins  ratio  proved  helpful  in  detecting  light 
images.  A  cutoff  value  of  .255  was  used. 

5.  Fill  factor:  Images  with  any  worthwhile  information  were  found  to  have  a  fill 
factor  of  at  least  two.  Otherwise,  they  were  either  completely  white  or  completely 
black. 

See  Table  3-1  for  a  summary  of  these  ratios.  It  was  found  that  by  applying  a  combination  of  these 
techniques,  most  of  the  bad  images  were  successfully  detected.  As  shown  in  Table  3-2,  only  two 
out  of  21 1  bad  images  were  falsely  accepted,  which  corresponds  to  a  false  acceptance  rate  of  less 
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than  1%.  However,  about  18  out  of  82  good  images  were  falsely  rejected,  which  corresponds  to  a 
false  reject  rate  of  about  22%.  This  means  that  if  most  of  the  incoming  data  were  of  good  quality, 
about  22%  of  them  would  still  be  rejected  and  require  manual  quality  assurance,  but  more  than 
99%  of  the  bad  drawings  would  be  correctly  rejected.  This  represents  a  significant  savings  as 
compared  to  a  100%  manual  quality  assurance  procedure.  The  detailed  results  for  this  set  are  listed 
in  Appendix  B. 


Table  3-1.  Acceptable  ratios  for  high  quality  criterion 


NO. 

RATIO 

LIMIT 

1 

Run  length  ratio 

<14 

2 

Approximate  black  orphan  ratio 

<0.02 

3 

Peak  tile  noise 

<156 

4 

Approximate  white  orphan  ratio 

<  0.255 

5 

Fill  factor 

>2 

Table  3-2.  Results  for  293  images  from  Data  Set  1 


Total  no.  of  images 

No.  of  images  acceptable  on  visual  inspection 

82 

No.  of  images  accepted  by  CADA  algorithms 

64 

No.  of  images  falsely  accepted 

2 

No.  of  images  falsely  rejected 

18 

Percentage  of  bad  images  falsely  accepted 

0.95 

Percentage  of  all  images  falsely  accepted 

0.68 

Percentage  of  good  images  falsely  rejected 

21.95 

Percentage  of  all  images  falsely  rejected 

6.14 

3.3.2  Image  Data  Set  2  Results 

As  described  in  Section  2.1.2,  a  representative  set  of  541  images  were  selected  from  a  total  of 
5,194  DSREDS/EDCARS  images.  The  main  aim  of  testing  this  set  was  to  see  how  well  the 
CADA  techniques  worked  with  the  existing  “live”  data.  The  criteria  for  acceptability  in  the  case  of 
this  data  were  readability  and  reproducibility  of  important  information  areas  within  an  image.  The 
presence  of  even  significant  amount  of  noise  or  fading  were  not  considered  sufficient  reasons  for 
rejection  if  the  important  parts  of  a  drawing  were  legible. 

,The  first  step  was  to  apply  the  “high  quality”  (Set  1)  combination  of  techniques  (as  described  in  the 
previous  subsection)  to  the  DSREDS/EDCARS  images.  Since  most  of  those  images  would  be 
considered  to  be  of  poor  quality  if  they  represented  new  CALS  data  delivered  to  the  government  by 
a  contractor,  a  very  high  number  of  them  were  rejected  by  this  combination,  as  summarized  in 
Table  3-3. 
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Table  3-3.  Results  on  applying  “high  quality  criterion”  to  Data  Set  2 


Total  no.  of  images 

541 

No.  of  images  acceptable  on  visual  inspection 

527 

No.  of  images  accepted  by  CADA  algorithms 

115 

No.  of  images  falsely  accepted 

0 

No.  of  images  falsely  rejected 

410 

Percentage  of  bad  images  falsely  accepted 

0 

Percentage  of  all  images  falsely  accepted 

0 

Percentage  of  good  images  falsely  rejected 

77.80 

Percentage  of  all  images  falsely  rejected 

75.79 

The  next  challenge  was  to  find  a  suitable  combination  that  would  accept  most  of  the 
DSREDS/EDCARS  images  that  were  found  acceptable  on  visual  inspection,  while  still  rejecting 
most  of  the  “bad”  images.  The  images  were  again  inspected  to  identify  some  features  that  would 
be  helpful  in  choosing  the  right  techniques.  Among  the  features  observed,  the  following  were  the 
most  striking. 

1 .  Border  noise  -  Many  images  had  a  lot  of  noise  along  the  borders.  It  was  either 
due  to  over-scanning  or  due  to  presence  of  banners  along  the  borders.  In  many 
cases,  the  actual  drawing  was  of  a  fairly  good  quality,  but  the  noise  along  the 
borders  was  so  overwhelming  that  it  caused  the  entire  image  to  get  a  poor  quality 
rating. 

2.  Lines  of  1-pixeI  width  -  Some  images  showed  presence  of  thin,  only  one  pixel 
wide  lines.  These  lines  would  normally  be  considered  as  unacceptable,  because 
they  do  not  confonn  to  ANSI  Y 14. 2M  standard  for  engineering  drawings  (as  j 
discussed  in  the  section  on  run  length  analysis).  However,  due  to  the  relaxed 
acceptability  standards  for  DSREDS/EDCARS  data,  many  such  drawings  were 
considered  acceptable. 

3.  Broken  lines  -  Many  images  had  lines  that  were  broken  in  many  places.  Again, 
those  images  would  most  likely  be  rejected  if  the  data  were  coming  from  a 
contractor,  but  many  of  them  were  deemed  acceptable  under  the  relaxed  standards. 

Due  to  the  overwhelming  presence  of  one-pixel  width  lines,  the  run  length  ratio  caused  a  large 
number  of  rejects.  Therefore,,  the  acceptable  value  for  this  ratio  was  raised  to  30,  but  the  results 
obtained  were  still  not  satisfactory.  So,  it  was  decided  to  discard  this  technique  altogether  for  this 
set  of  data.  The  approximate  white  orphan  ratio  and  peak  tile  noise  indexes  were  also  discarded  for 
similar  reasons.  To  eliminate  the  effect  of  border  noise,  border  clippings  of  different  sizes  (0.5 
inch,  1  inch,  1.5  inches)  were  tried.  It  was  found  that  clipping  1.5  inches  (300  pixels)  off  the 
border  was  the  most  effective  way  to  eliminate  the  impact  of  border  noise  upon  the  results. 
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Eventually,  the  best  results  were  obtained  by  using  the  approximate  black  orphan  ratio  with  an 
acceptable  value  of  less  than  0.06  and  a  fill  factor  with  an  acceptable  value  of  greater  than  0.5 
(both  of  these  values  were  computed  after  ignoring  1.5  inches  along  all  the  four  borders).  These 
ratios  arc  shown  in  Table  3-4.  On  applying  these  techniques  to  the  set  of  541 DSREDS/EDCARS 
images,  it  was  determined  that  out  of  the  527  visually  acceptable  images,  57  were  falsely  rejected 
and  out  of  14  bad  images,  only  one  was  falsely  accepted.  This  meant  that  out  of  the  total  set  of 
541, 0.18%  of  the  images  were  falsely  accepted  and  10.53%  were  falsely  rejected.  These  results 
are  summarized  in  Table  3-5.  The  detailed  results  are  listed  in  Appendix  B. 


Table  3-4.  Acceptable  ratios  for  “low  quality”  criterion 


NO. 

RATIO 

■ilAIUlHHH 

1 

Approximate  orphan  ratio 
(after  clipping  1 .5  inches 
along  all  four  borders) 

<0.06 

2 

Fill  Factor 
(after  clipping 

1.5  inches  along  all 
four  borders) 

>  0.5 

Table  3-5.  Results  on  applying  “low  quality  criterion”  to  Data  Set  2 


Total  no.  of  images 

541 

No.  of  images  acceptable  on  visual  inspection 

527 

No.  of  images  accepted  by  CADA  algorithms 

468 

No.  of  images  falsely  accepted 

1 

No.  of  images  falsely  rejected 

57 

Percentage  of  bad  images  falsely  accepted 

7.14 

Percentage  of  all  images  falsely  accepted 

Percentage  of  good  images  falsely  rejected 

Percentage  of  all  images  falsely  rejected 

Both  the  “high  quality”  and  “low  quality”  techniques  were  applied  to  the  entire  set  of  5,194 
DSREDS/EDCARS  images  to  get  a  measure  of  the  resulting  number  of  rejects.  Due  to  the  fact  that 
all  of  the  images  were  not  visually  examined,  the  number  of  false  accepts  and/or  false  rejects  could 
not  adequately  be  determined.  The  main  goal  was  to  get  a  feel  for  the  total  number  of  rejected 
images.  It  was  found  that  the  “high  quality”  combination  resulted  in  a  rejection  rate  of  66.71% 
while  the  “low  quality”  combination  resulted  in  a  rejection  rate  of  9.28%.  These  results  are 
summarized  in  Tables  3-6  and  3-7. 
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Table  3-6.  Results  on  applying  “high  quality  criterion”  to  the  entire  set  of  DSREDS/EDCARS  images 


Total  no.  of  images 

5194 

No.  of  images  accepted  by  CADA  algorithms 

1729 

No.  of  images  rejected  by  CADA  algorithms 

3465 

Percentage  of  rejects 

66.71 

Table  3-7.  Results  on  applying  “low  quality  criterion”  to  tbe  entire  set  of  DSREDS/EDCARS  images 


Total  no.  of  images 

5194 

No.  of  images  accepted  by  CADA  algorithms 

4753 

No.  of  images  rejected  by  CADA  algorithms 

441 

Percentage  of  rejects 

9.28 

3.4  SUMMARY  OF  TEST  RESULTS 

The  following  paragraphs  summarize  the  results  of  the  image  quality  analysis  testing. 

1.  A  combination  of  five  techniques  was  identified  for  a  high  level  of  quality  control 
standards.  Using  this  combination,  one  could  obtain  a  false  accept  (FA)  rate  of  less 
than  1%  and  a  false  reject  (FR)  rate  of  22%.  In  a  typical  procurement  received  from 
a  contractor  it  would  be  expected  that  the  good  data  would  exceed  90%.  If  a  1000 
image  batch  was  analyzed  for  acceptance  and  there  were  100  bad  images,  the 
combined  techniques  as  tested  would  falsely  accept  one  bad  image  and  falsely  reject 
900  X  0.22  or  20  images.  These  20  would  be  viewed  and  the  rejection  overridden. 

2.  The  application  of  the  above  techniques  to  DSREDS/EDCARS  data  resulted  in  a 
very  high  rejection  rate,  because  most  of  that  data  would  be  considered  to  be  of 
poor  quality  if  they  were  new  data  coming  from  a  contractor.  To  accept  that  data, 
some  of  the  “high  quality”  techniques  had  to  be  dropped,  and  for  others  the  quality 
parameters  had  to  substantially  relaxed. 

So,  the  selection  of  techniques  and  setting  of  parameters  for  image  quality  depends 
greatly  on  the  strictness  of  quality  control  standards.  Due  to  this  reason,  the  CADA 
tools  for  field  testing  should  allow  the  operators  to  configure  the  techniques  and 
parameters,  depending  on  the  characteristics  of  the  data  and  strictness  of  quality 
control. 

3.  The  quality  analysis  techniques  can  be  most  effectively  used  if  the  quality  control  is 
based  on  objective  criteria  like  conformance  to  ANSI  standards  for  line  widths, 
absence  of  noise,  lack  of  fading,  etc.  If  the  quality  control  is  subjective  and  is 
based  more  on  interpretation  of  the  contents  rather  than  on  the  criteria  mentioned 
above,  these  techniques  will  not  give  a  good  performance. 
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SECTION  4 

IDENTIFICATION  RECOGNITION  ANALYSIS  AND  TESTING 


4.1  TEST  PROCEDURES 

Identification  recognition  is  the  automated  process  of  extraction  and  recognition  of  major  drawing 
identification  data  from  engineering  drawings.  The  identification  data  of  each  engineering  drawing 
includes:  drawing  number,  sheet  number,  size,  FSCM  number,  revision  number,  etc.  This 
section  describes  the  tools  selection  and  test  procedures  performed  for  the  identification  recognition 
of  engineering  drawings  during  this  task. 

Step  1:  The  first  step  to  perform  the  identification  recognition  test  was  to  select  the 
commercially  available  tools  for  this  task.  The  previous  phase  of  the  CADA 
technical  report  (see  Technical  Report,  Testing  Techniques  for  Data  Acceptance 
Procedures,  12  July  1991),  listed  available  tool  sets  to  perform  the  identification 
recognition  test.  Potential  candidates  were  selected  from  this  list.  Each  interested 
vendor  was  given  approximately  200  title  block  images  in  ID  Data  Set  1  which  was 
produced  and  obtained  as  described  in  Section  2.2.  The  vendors  were  asked  to 
provide  their  test  results  based  on  the  provided  ID  Data  Set  1  Data.  Four  vendors 
were  selected  to  participate  in  the  tests  for  steps  2  and  3. 

Step  2:  The  second  step  in  the  test  process  was  to  test  vendors’  tools  in  the  Army  CTN  test 
bed  at  the  CTC  using  ID  Data  Set  2.  A  specification  for  the  needed  Identification 
Recognition  tool  was  constructed  and  given  to  the  four  vendors.  The  vendors’ 
tools  which  were  commercially  available  were  allowed  to  be  modified  as  necessary 
to  tailor  their  products  to  meet  the  given  specification.  The  ID  Recognition 
Requirement  Specification  is  included  in  Appendix  D.  The  tools  provided  by  the 
vendors  were  installed  in  the  CTC  facilities  in  a  controlled  environment  for  this  part 
of  the  test.  ID  Data  Set  2  used  in  this  part  of  the  test  was  different  fi’om  ID  Data  Set 
1  that  was  used  in  the  Step  1  test  at  vendors’  sites.  This  was  done  to  ensure  that 
the  Step  2  test  data  set  had  never  been  used  for  neural  network  character  recognition 
training  by  the  vendors.  The  ID  Data  Set  2  was  produced  and  obtained  as  described 
in  Section  2.2. 

Step  3:  The  third  step  in  the  test  process  was  to  test  the  same  software  as  in  Step  2  on  ID 

Data  Set  3.  ID  Data  Set  3  has  better  image  quality  then  ED  Data  Set  2.  It  is  expected 
that  the  raster  image’s  quality  of  data  procured  using  the  CALS  standards  will  be  of 
higher  quality  than  the  existing  data  of  Data  Set  2.  The  objective  was  to  test  the 
vendor  software  with  data  that  closely  parallel  data  that  meet  the  CALS  standards 
and  specifications. 

See  Sections  4.3  and  4.4  for  the  analysis  and  summarization  of  these  tests. 
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4.2  IDENTIFICATION  RECOGNITION  TECHNIQUES 

Identification  recognition  techniques  described  below  were  used  by  the  tools  tested  to  perform  the 
automated  process  of  extraction  and  recognition  of  major  drawing  identification  data  from 
engineering  drawings.  Currently,  the  identification  process  of  each  engineering  drawing  is 
performed  manually  at  the  repository  sites.  If  this  data  can  be  processed  electronically  using 
modem  image  and  character  recognition  technology,  rather  than  the  present  labor  intensive 
manual/visual  validation  method,  a  time  savings  can  be  obtained. 

Character  recognition  is  one  component  of  the  ID  recognition  automation  solution.  To  recognize  a 
drawing’s  identification  data,  the  software  must  locate  the  text  image  field  and  separate  it  fi’om 
other  data  in  the  drawing.  Next,  it  must  recognize  the  characters  in  the  field,  verify  the  information 
and  format  it  to  the  correct  output  specifications  (see  Appendix  D).  Many  different  technologies 
and  approaches  can  be  used  to  achieve  this  objective.  Products  provided  by  various  vendors  using 
different  technologies  were  tested  and  assessed  for  the  feasibility  of  the  CADA  identification 
recognition. 

The  identification  recognition  process  for  the  CADA  task  consists  of  three  independent  stages: 
preprocessing,  character  recognition,  and  post  recognition  validation.  These  stages  are 
independent  functions;  yet  they  are  related  in  the  sense  that  they  may  affect  the  other  functions’ 
performance  and  accuracy. 

4.2.1  Preprocessing 

The  first  step  in  the  CADA  identification  recognition  process  is  to  look  for  the  title  block  contained 
in  the  raster  image  of  the  engineering  drawing.  This  entails  locating  the  title  block  image  area  and 
all  of  the  text  fields  (and  their  data)  within  that  area,  and  then  extracting  the  entire  text  image  for 
character  recognition.  Once  the  appropriate  text  image  data  are  isolated,  they  are  segmented  into 
characters  for  recognition.  This  process  of  searching  for  the  title  block  area  and  the  extraction  of 
the  appropriate  text  image  data  is  call  preprocessing.  The  preprocessing  function  for  the  ID 
recognition  requires  a  specialized  software  tool.  No  commercial  off-the-shelf  (COTS)  products 
exist  at  the  present  time  which  can  do  the  preprocessing  of  the  title  block  area. 

The  preprocessing  function  is  very  similar  to  the  forms  processing  used  in  businesses  such  as  the 
financial  community  (e.g.,  IRS  forms  and  check  processing),  where  the  data  being  processed 
exists  within  a  predetermined  area  and  structure  (e.g.,  checks  and  IRS  1040s).  It  is,  therefore, 
relatively  easy  for  the  computer  to  extract  the  necessary  information.  Unfortunately,  the  same 
technology  used  in  this  area  encounters  many  difficulties  when  it  is  applied  to  engineering  title 
block  processing,  due  to  the  inherent  non-standard  structure  of  the  data,  and  the  area  in  which  that 
data  resides. 

A  standard  does  exist  for  engineering  drawings  (ANSI  Y14.1-1975,  American  National  Standard 
for  Engineering  Drawings  and  Related  Documentation  Practices)  published  by  the  American 
Society  of  Mechanical  Engineering.  Many  of  the  engineering  drawings  that  were  received  fix)m  the 
repositories  do  not  follow  this  standard,  especially  those  drawings  completed  before  the  standard 
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was  published.  Even  many  of  the  drawings  that  mostly  comply  with  the  standard  contain  minor 
variations,  such  as  a  slight  difference  in  the  width  and  length  of  the  SIZE  box.  In  tradidonal  forms 
processing,  a  template  is  created  for  each  variation.  This  may  not  be  plausible  for  engineering 
drawings  due  to  the  many  minor  variations  that  exist  within  the  data  held  at  each  repository. 

The  preprocessing  depends  on  the  following  technologies  to  extract  text  image  from  the  title 
blocks. 

4.2.1.1  Line  Tracing 

In  the  line  tracing  method,  the  procedure  follows  the  pixel  lines  horizontally  and  vertically.  The 
positions  of  the  text  boxes  are  calculated  by  the  intersection  of  horizontal  and  vertical  lines.  Once 
the  text  boxes  are  located,  the  text  image  in  each  box  can  be  identified  by  the  histogram  of  the 
number  of  pixels.  This  approach  has  trouble  with  broken  lines,  therefore,  line  enhancement 
algorithms  are  needed  to  identify  the  text  boxes  in  poor  quality  drawings. 

4.2.1.2  Corner  Matching 

This  procedure  locates  all  the  matching  corners  of  a  box  (i.e.,  it  must  locate  the  normal  four 
comers  of  a  box  before  the  area  can  be  identified  as  a  box).  Because  line  tracing  is  not  involved, 
this  method  does  not  have  the  problem  of  broken  lines  in  the  recognition  process.  However,  this 
approach  is  very  sensitive  to  the  objects  around  and  adjacent  to  the  comers  it  attempts  to  find, 
therefore,  may  report  false  comers.  It  will  also  miss  comers  if  the  image  is  tilted  or  askew. 

4.2.1.3  Structure  Recognition 

Stmcture  recognition  uses  neural  network  technology  to  recognize  the  position  and  shape  of  the 
boxes  in  the  title  area  of  engineering  drawings.  A  separate  neural  network  can  be  trained  to 
recognize  the  pattern  of  the  title  block.  Each  field  within  the  title  block  can  then  be  recognized  and 
labeled  by  the  network,  and  text  information  extracted  according  to  field  identification.  Obviously, 
this  method  can  provide  a  more  flexible  and  tolerant  system  to  accommodate  the  minor  variations  in 
the  title  block  formats.  However,  additional  neural  network  training  would  be  needed  to  recognize 
any  major  pattern  change  in  that  area. 

4.2.1.4  Pure  Text  Extraction 

This  approach  ignores  all  forms  stmcture,  and  only  recognizes  and  extracts  all  text  images  in  the 
input  data.  After  each  field  is  recognized,  it  needs  to  be  identified  and  correlated  based  on  the 
content  of  the  recognized  word.  Here  the  problem  is  that  the  correlation  and  identification 
processes  can  be  erroneous  and  ambiguous.  For  example,  the  size  of  the  drawing  may  be 
identified  only  by  a  field  name  (SIZE);  that  particular  field  normally  contains  a  single  alpha 
character  (e.g.,  B).  However,  there  is  no  guarantee  that  the  recognized  field  in  this  case  is  the 
SIZE  field,  because  there  may  be  other  single  character  fields  on  the  drawing.  Because  the 
stmcture  information  is  not  considered  during  extraction,  as  is  evident  from  the  previous  example, 
this  method  is  not  completely  reliable. 
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4.2.2  Character  Recognition 

The  accuracy  of  character  recognition  technology  has  improved  from  Optical  Character  Recognition 
(OCR)  to  Intelligent  Character  Recognition  (ICR).  OCR  techniques  rely  on  one  or  two  rule-based 
methods  for  reading  machine  printed  characters:  template  matching  or  feature  extraction  (by  fonts), 
while  ICR  uses  trained  neural  networks  which  are  capable  of  recogni2dng  handwritten  and 
hand-printed  alphanumerics.  It  has  been  concluded  in  the  previous  CADA  technical  report 
{Technical  Report,  Testing  Techniques  for  Data  Acceptance  Procedures,  12  July  1990)  that  OCR 
technology  will  not  produce  enough  accuracy  in  recognizing  hand-printed  engineering  drawing 
data  for  the  CADA  ID  recognition.  The  selection  of  vendor  products  for  the  character  recognition 
engine  of  this  test  was  focused  on  only  the  products  using  ICR  techniques. 

4.2.2. 1  Neural  Network  Technology 

All  ICR  technologies  are  based  upon  neural  network  technology.  The  accuracy  of  a  neural 
network  is  very  much  dependent  upon  the  training  that  the  neural  network  receives,  and  the  target 
objects  to  be  recognized.  For  example,  some  neural  networks  are  trained  to  recognize  handwritten 
zip  codes,  while  others  are  trained  to  recognize  hand-printed  capital  letter  characters.  These  two 
networks  cannot  be  interchanged  because  the  accuracy  of  the  neural  network  decreases  if  die  target 
object  does  not  belong  to  the  particuku"  category  for  which  it  was  trained.  A  well  trained  character 
recognition  engine  can  separate  and  distinguish  similar  chiu’acters,  such  as  the  difference  between 
“5”  and  “S”  in  a  manner  similar  to  the  human  brain.  In  the  case  of  characters  that  look  exactly 
alike,  such  as  “0”  (zero)  and  the  letter  “O,”  there  is  no  way  for  the  neural  network  to  distinguish 
these  without  further  context  information.  The  postprocessing  stage  was  designed  precisely  for 
such  instances. 

Most  vendors’  neural  networks  were  originally  trained  to  recognize  handwritten  characters.  The 
input  for  these  networks’  training  basically  came  from  the  National  Institute  of  Standards  and 
Technology  (NIST)  test  and  training  data,  plus  each  vendor’s  own  sources  of  handwritten 
characters.  In  the  future,  neural  network  training  for  the  recognition  of  hand-printed  characters 
may  need  to  be  enhanced  (for  any  selected  vendor’s  product)  to  create  a  fully  reliable  production 
version  of  the  engine  for  CADA  at  future  test  sites.  Some  of  the  selected  vendors  have  started  to 
incorporate  such  additional  preliminary  handwritten  character  training  into  the  neural  networks  that 
were  delivered  for  the  tests  in  this  report. 

4.2.2.2  Character  Image  Segmentation 

Neural  network  algorithms  have  proven  useful  for  the  recognition  of  individual,  segmented 
characters.  However,  their  recognition  accuracy  has  been  limited  by  the  accuracy  of  the  underlying 
segmentation  algorithm.  Segmentation  is  the  method  to  cut  and  separate  the  text  image  of  each 
individual  character  in  a  string  for  recognition.  Obviously,  it  is  a  very  difficult  task  for  the 
computer  to  decide  where  the  start  and  end  points  of  a  character  are,  especially  when  the  characters 
touch  each  other,  or  are  connected  to  a  boundary  line.  Even  though  most  of  the  commercial  neural 
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networks  can  be  retrained  to  recognize  the  different  kinds  of  letters  on  engineering  drawings  to 
increase  recognition  accuracy,  it  is  more  important  to  evaluate  whether  the  selected  vendor’s 
recognition  engine  has  a  good  segmentation  algorithm  for  our  application. 

Qirrently,  preprocessing  and  segmentation  algorithms  encounter  difficulty  if  the  drawings  and 
characters  are  touching,  broken,  or  noisy.  The  accuracy  of  the  identification  recognition  results 
was  very  high  for  the  engineering  drawings  complying  with  standard  format  and  with  good  quality 
raster  image.  The  CADA  acceptance  criteria  shall  reject  those  engineering  drawings  where 
identification  of  the  title  block  can  not  be  successfully  processed. 

4.2.3  Post  Recognition  Validation 

Once  a  character  image  is  isolated  and  possible  winning  target  candidates  are  selected  by  the 
recognition  process,  the  final  recognition  selection  can  be  decided  normally  by  the  highest 
confidence  level  of  each  character.  The  candidate  character’s  confidence  level  is  one  of  the  ordinary 
output  data  for  most  commercially  available  intelligent  character  recognition  engines.  There  are 
other  factors  that  can  be  used  to  provide  better  selection,  if  the  context  environment  of  the  character 
is  known  and  defined.  For  example,  if  the  character  to  be  recognized  is  a  part  of  a  word,  then  the 
final  outcome  will  be  validated  against  words  in  a  dictionary.  Characters  creating  a  correedy 
spelled  word  will  be  picked,  despite  the  low  confidence  level  calculated  by  the  recognition  engine. 
The  predefined  rule  may  include  whether  it  is  a  field  of  alpha  or  numerical  characters,  or  by  the 
number  of  characters  present  in  the  field,  etc.  Therefore,  post  recognition  validation  can  add 
additional  heuristics,  and  new  rules  can  be  defined  in  advance  to  influence  the  final  character 
selection  decision. 

Postprocessing  is  very  much  dependent  on  a  particular  form  and  its  application,  ttice  we  can 
successfully  correlate  a  box  in  the  title  area  to  the  context  environment  of  the  image  to  be 
recognized,  the  accuracy  of  the  resultant  recognition  can  be  significandy  improved.  There  is  a 
significant  amount  of  intelligent  software  which  must  be  implemented  in  ordw  to  do  the 
postprocessing.  Due  to  the  current  test  schedule,  only  a  very  limited  amount  of  work  was  done  in 
the  postprocessing  area.  The  efforts  in  this  area  are  to  be  enhanced  during  the  integration  test 
phase  in  order  to  improve  recognition  accuracy. 

4.3  TEST  RESULTS 

The  identification  recognition  tests  were  done  on  Sun  Sparc  Workstations  and  PCs.  Two  out  of 
the  four  vendors  participated  in  the  test  using  Sun  Sparc  workstation,  and  the  other  two  use  PC. 
The  Sun  Sparc  Workstations  are  configured  with  Sun  OS  Version  4.1.1  and  OpenWindows 
Version  3.0.  The  PCs  are  configured  with  Intel  80386  or  80486  chip  and  Microsoft  Windows 
Version  3.0. 


For  each  engineering  drawing  file  tested  for  ID  recognition,  four  fields  were  to  be  recognized: 
drawing  number,  sheet  number,  size,  and  FSCM  number.  Detailed  test  results,  for  each  individual 
engineering  drawing  by  vendor  name  are  included  in  a  tabular  form  in  Appendix  C.  In  Appendix 
C,  the  original  value  and  the  recognized  value  for  each  field  of  all  tested  engineering  drawings  are 
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both  listed  in  the  table  for  side-by-side  comparison.  All  correctly  recognized  fields  and  characters 
were  counted  and  the  percentage  of  correct  recognition  ratio  were  calculated. 

The  following  paragraphs  summarize  the  test  results  of  test  Steps  1, 2  and  3  as  delineated  in 
Section  2.2. 

4.3.1  ID  Data  Set  1  Used  for  Tools  Selection 

The  first  step  to  perform  the  identification  recognition  test  was  to  select  the  commercially  available 
tools  to  perform  the  identification.  The  following  is  the  preliminary  list  of  available  vendors 
initially  selected  as  the  potential  candidates  to  perform  the  identification  recognition  test: 

AT&T, 

HNC, 

Nestor, 

OCR  systems/Non-Linear  Technology  (NTI), 

VisionShape, 

MCC, 

Symbus, 

CAERE, 

Calera, 

Datacap, 

Ektron, 

NYNEX, 

OCRON,  and 
Recognitto. 

Some  of  the  above  vendors  had  participated  in  the  preliminary  test  which  was  conducted  for  the 
previous  phase  of  the  CADA  technical  report  (see  Technical  Report,  Testing  Techniques  for  Data 
Acceptance  Procedures,  12  July  1991).  Each  vendor  was  given  the  approximately  200  title  block 
images  in  the  ID  Data  Set  1  which  were  produced  and  obtained  as  described  in  Section  2.2.  The 
vendors  were  asked  to  provide  their  test  results  based  on  the  provided  ID  Data  Set  1  Data.  ID  Data 
Set  1  were  used  by  vendors  for  that  neural  network  training  as  well  as  for  their  initial  feasibility 
test. 

As  the  result  of  the  first  step,  four  vendors  were  selected  based  on  their  test  performance.  The  four 
companies  that  were  selected  and  joined  the  follow-on  tests  were  MCC,  Nestor,  OCR/NTI  and 
VisionShape.  The  other  companies  were  eliminated  due  to  one  or  more  of  the  following  reasons: 

•  did  not  have  ICR  recognition  technology  that  is  necessary  to  provide 
required  recognition  accuracy  to  satisfy  our  needs; 

•  could  not  have  their  product  ready  to  meet  our  test  schedule; 

•  did  not  have  preprocessing  capability  to  utilize  their  ICR  technology; 

•  would  not  participate  due  to  lack  of  short  term  incentives;  or 

•  needed  specialized  hardware. 
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4.3.1.1  Description  of  Tools  Tested 

This  paragraph  describes  the  tools  selected  for  testing.  The  four  tools  selected  are: 

•  NestorReader™  version  1 .0  by  Nestor.  NestorReader  accepts  binary  images  in 
both  TIFF  and  PCX  formats,  segments  and  recognizes  both  constrained  and 
unconstrained  touching  characters,  and  does  its  own  image  compression  and 
decompression.  Nestor  added  pure  text  extraction  to  the  front-end  of  the 
NestorReader  software  for  the  preprocessing  step  of  the  CADA  tests  so  that  the 
identified  candidate  character  string  image  could  be  recognized  by  NestorReader. 
Nestor  has  specially  trained  neural  networks  for  different  types  of  characters,  such 
as  machine-printed,  hand-written,  alphabetic,  numeric,  etc.  Character  recognition 
capability  was  improved  by  adding  additional  segmentation  algorithms  and  further 
neural  network  training  of  the  hand-printed  characters  on  the  engineering  drawings. 

•  ReadRight  OCR  by  OCR  Systems/NTI.  NTI's  product  performs  reasonably  well 
and  at  an  acceptable  speed  without  additional  hardware  boards  to  accelerate  the  PC. 
It  has  the  capability  of  recognizing  handwritten  characters  of  alphabetic,  numeric,  or 
mixed  type.  It  has  the  capability  of  distinguishing  mixed  alphabetic  and  numeric 
characters  and  has  a  better  recognition  accuracy  than  any  other  software  that  was 
tested.  NTI  uses  line  tracing  techniques  to  locate  the  boxes  in  the  title  block  area  in 
order  to  perform  the  preprocessing  function. 

•  Microelectronics  and  Computer  Technology  Corporation(MCC)  used  a  supervised 
learning  algorithm  (back  propagation),  developed  in  their  laboratory  environment, 
to  recognize  and  segment  the  hand-printed  numerical  characters  with  overlapping. 
They  do  not,  however,  have  the  capability  of  preprocessing.  The  front-end  text 
image  extraction  was  performed  by  Act  Laboratory,  Altamonte  Springs,  FL.  The 
Act  Laboratory's  preprocessing  algorithm  used  comer  matching  technology. 

•  VisionShape,  Inc.,  developed  AUTOCLASS  which  performs  automatic  form 
processing  on  an  IBM  PC  or  PC  compatible  machines.  AUTOCLASS  uses  neural 
network  technology  to  recognize  and  identify  the  structure  and  each  field  within  a 
form,  in  contrast  to  the  other  companies,  which  use  image  technology  to  locate  the 
fields  in  a  form  by  lines  and  boxes.  Visionshape's  primary  business  area  is  in  form 
processing  while  the  other  three  selected  companies  are  in  the  area  of  character 
recognition.  Thus,  the  product  provided  by  VisionShape  exhibits  a  different 
approach.  Some  of  the  other  companies  involved  in  this  test  have  suggested  that 
they  will,  in  the  future,  experiment  with  preprocessing  similar  to  Visionshape's 
approach.  The  VisionShape  experience  and  results  have  provided  a  good  lesson  on 
this  type  of  approach  for  form  stmcture  recognition. 
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4.3.2  ID  Data  Set  2  Results 

The  statistics  of  the  test  results  were  calculated  by  two  major  indicators:  the  percentage  of  correct 
characters  recognized  and  the  percentage  of  correct  fields  recognized  There  are  four  fields  in  an 
engineering  drawing  title  block  to  be  identified:  drawing  number,  sheet  number,  size,  and  FSCM 
number.  Table  4-1  shows  the  test  results  for  the  Data  Set  2  test  data.  There  is  a  total  of  73  files  of 
engineering  drawings  selected  in  the  Data  Set  2.  It  includes  254  fields  which  consist  of  1 191 
characters,  in  total,  to  be  identified  for  this  test.  Visionshape’s  software  failed  to  process  some  of 
the  Data  Set  2  files.  Those  files  are  excluded  from  the  results  statistics. 

Table  4-1.  ID  Data  Set  2  Results 


Total  Fields  Tested 
Field  Recognized 

Percentage  Correctly  Recognized  Fields 
Total  Characters  Tested 
Characters  Recognized 
Percentage  Correctly  Recognized  Characters 


Nestor 

NTI 

!  VisionShaoe  1 

254 

254 

254 

00 

oo 

89 

90 

62 

38 

35% 

35% 

24% 

20% 

1191 

1191 

1191 

799 

638 

580 

542 

344 

54% 

49% 

46% 

43% 

4.3.2.1  Overall  Recognition  Results 

The  overall  recognition  test  results  for  the  Identification  Recognition  were  in  the  range  of  43%  to 
54%  for  correct  characters  recognized  and  20%  to  35%  for  correct  fields  recognized. 

Nestor  has  the  highest  rating  (54%)  of  overall  character  recognition.  Both  Nester  NTI  have  the 
highest  rating  (35%)  if  counting  each  field  as  the  basic  recognition  unit. 

The  overall  recognition  percentage  was  lower  than  expected.  One  reason  for  the  low  percentage 
was  that  the  vendors’  software  did  not  perform  well  in  the  preprocessing  stage  to  capture  the  right 
text  image  that  can  be  recognized. 

It  is  a  major  challenge  to  locate  the  title  block  within  the  raster  image  of  an  engineering  drawing. 
The  data  set  used  in  this  test  does  not  have  good  raster  image  quality  (there  are  broken  lines  that 
make  the  identification  of  the  border  of  a  box  difficult).  It  is  believed  that  with  a  data  set  of  better 
raster  image  quality,  the  recognition  percentage  can  be  improved  considerably.  ID  Data  Set  3  was 
selected  with  better  quality  of  raster  data  and  the  next  step  (Step  3)  of  the  test  procedure  was 
conducted  to  prove  the  concept  of  this  idea  as  described  in  4.3.3. 

4.3.2.2  Detailed  Analysis 

Nestor  has  provided  the  tool  with  the  most  stable  software  of  the  four  products  tested.  Its  final 
character  recognition  capability  was  specially  tailored  to  recognize  numerics  and  machine-printed 
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alphabetic  for  the  type  of  characters  used  in  the  engineering  drawings.  Due  to  the  pure  text 
extraction  technique  used  in  the  preprocessing,  an  additional  software  capability  must  be  added  at  a 
later  date  to  automatically  correlate  the  recognized  characters  with  their  original  fields. 

NTI's  recognition  engine  is  the  fastest  of  the  products  tested,  and  uses  the  least  computer 
resources.  Their  neural  network  and  recognition  engine  training  method  have  some  basic 
differences  with  others,  as  NTI  claims.  NTI  uses  line  tracing  techniques  to  find  boxes  in  the  title 
block  area  to  perform  the  preprocessing  function.  This  method  does  not  work  well  for  drawings 
with  poor  raster  line  quality.  For  the  drawings  with  good  raster  line  quality,  NTT’s  product  has  the 
highest  recognition  ratio. 

MCC  uses  ACT  laboratory's  preprocessing  algorithm  that  uses  corner  matching  technology  to  find 
text  images  in  the  boxes.  The  problem  with  this  approach  is  that  it  is  very  sensitive  to  the  other 
objects  around  the  corners,  such  as  raster  image  noise  and  footnotes  near  the  comers,  etc.  The 
preprocessing  software  used  for  the  test  is  in  the  development  stage,  and  is  not  very  stable  when 
processing  bad  quality  image  data;  processing  time  is  also  slow. 

VisionShape  trained  its  neural  network  Autoclass  to  locate  the  boxes  in  engineering  drawings. 
Autoclass  performs  recognition  of  fields  by  line  location  and  relative  position  matching  of  these 
intersecting  lines.  Like  the  line  tracing  method,  processing  and  recognition  of  broken  and  poor 
quality  lines  was  a  challenge.  Visionshape’s  current  software  works  well  only  for  the  engineering 
drawings  with  standard  format.  There  software  is  not  stable  to  process  title  blocks  that  may 
deviate  fix)m  standard  formats. 

4.3.3  ID  Data  Set  3  Results 

The  statistics  of  the  test  results  were  calculated  by  two  major  indicators:  the  percentage  of  correct 
characters  recognized  and  the  percentage  of  correct  fields  recognized,  in  a  similar  fashion  to  the 
Data  Set  2  test  results.  Table  4-2  shows  the  test  results  for  the  Data  Set  3  test  data.  There  is  a  total 
of  88  files  of  engineering  drawings  selected  in  the  Data  Set  3.  It  includes  311  fields  which  consist 
of  1474  characters  in  total  to  be  identified.  VisionShape’s  software  failed  to  take  some  of  the  Data 
Set  3  files  as  input.  The  Visionshape’s  statistical  results  exclude  those  files. 

Table  4-2.  ID  Data  Set  3  Results 


Nestor 

NTI 

VisionShape 

Total  Fields  Tested 

311 

311 

91 

Field  Recognized 

177 

280 

34 

Percentage  Correctly  Recognized  Fields 

57% 

90% 

62% 

37% 

Total  Characters  Tested 

1474 

1474 

1474 

451 

Characters  Recognized 

990 

1405 

1229 

228 

Percentage  Correctly  Recognized  Characters 

67% 

95% 

83% 

51% 
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4.3.3.1  Overall  Recognition  Results 

The  overall  recognition  test  results  for  the  Identification  Recognition  were  in  the  range  of  51%  to 
90%  for  the  percentage  of  correct  character  recognized  and  57%  to  90%  for  the  correct  fields 
recognized.  The  ID  Data  Set  3  test  results  are  better  than  the  ID  Data  Set  2  test  results  for  all  the 
four  tools.  The  same  version  of  software  for  each  vendor  was  used  for  both  the  tests.  The  only 
difference  between  the  two  tests  were  the  different  data  sets  used;  however,  this  data  is  more 
representative  of  the  quality  of  CALS  data  expected  to  be  received  from  contractors.  The  test 
results  show  marginal  improvements  for  Nestor’s  and  Visionshape’s  software,  but  they  show  a 
two  fold  increase  in  the  recognition  percentage  of  NTI’s  and  MCC’s  software.  The  reasons  are 
presented  in  the  Detailed  Analysis  Section  below. 

NTI  has  the  highest  rating  of  overall  character  and  field  recognition  percentages  of  95%  and  99% 
respectively.  TTie  good  recognition  percentages  were  at  the  level  expected  due  to  the  better  image 
quality  in  the  ID  Data  Set  3.  The  improved  image  allows  better  preprocessing  to  locate  the  boxes 
and  the  text  images  in  the  boxes  to  generate  a  candidate  image  to  perform  character  recognition. 

The  improved  raster  image  actually  does  not  change  the  recognition  of  the  capability  as  much  as  the 
preprocessing  capability  because  there  was  no  change  in  the  nature  of  the  printing  style  of 
characters. 

4.3.3.2  Detailed  Analysis 

For  Nestor’s  tool,  this  test  data  set,  with  enhanced  quality  of  lines  and  boxes,  does  not  help  much 
in  increasing  Nestor’s  preprocessing  or  recognition  capability.  Because  of  the  fact  that  Nestor’s 
preprocessing  does  not  locate  tlie  lines  and  boxes  to  identify  the  text  field  images,  the  enhanced 
raster  image  does  not  improve  the  Nestor’s  recognition  capability.  The  better  raster  quality  image 
will  help  their  preprocessing  capability  at  a  later  date,  to  automatically  correlate  the  recognized 
characters  with  the  original  fields. 

NTI's  recognition  percentage  had  the  most  improvement  in  the  test  results  for  this  set  as  compared 
to  the  ID  Data  Set  2.  With  the  improved  line  and  box  quality,  the  preprocessing  step  of  NTI’s 
software  was  able  to  locate  more  accurately  the  text  images  to  be  recognized  by  its  recognition 
engine.  The  high  recognition  rate  of  NTI’s  results  did  demonstrate  the  superior  recognition 
algorithms  used  in  their  products.  The  recognition  rate  of  their  recognition  engine  was  claimed  by 
NTI  at  95%  to  99%  level;  however,  the  test  results  were  at  a  90%  tom95%  level  for  this  test  case. 

MCC’s  test  results  show  the  same  improvements  as  NTI’s  due  to  the  same  reason  that  MCC’s 
recognition  engine  was  able  to  locate  the  target  text  image  more  accurately  for  recognition.  The 
MCC’s  software  is  in  development  and  runs  very  slowly.  MCC  is  investigating  incorporating  their 
algorithm  into  an  LSI  chip  for  high  performance. 

Visionshape’s  test  was  mainly  for  studying  the  feasibility  of  performing  the  preprocessing  using 
neural  network  technology.  Character  recognition  was  not  one  of  Visionshape’s  strong  points. 

Due  to  the  unstable  condition  of  Visionshape’s  software,  the  test  is  not  conclusive.  For  the  whole 
test  suite,  less  than  40%  of  data  files  were  able  to  be  processed. 
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4.4  SUMMARY  OF  TEST  RESULTS 

The  following  paragraphs  summarize  the  results  of  the  identification  recognition  testing. 

1.  With  the  current  pre-processing  and  recognition  technology,  the  achieved 

percentage  of  successful  recognition  rate  on  good  data  is  above  90%.  Among  the 

four  tools  tested,  NTI  has  the  best  test  results  followed  by  MCC’s  test  result  as  a 
close  second. 


2.  In  addition  to  the  recognition  rate,  some  other  fetors  were  considered  and  a 

comparison  was  made  for  the  four  tools  tested.  Table  4-3  shows  the  results  of 
overall  comparison  by  the  factoK  considered.  The  indicators  used  in  the  Table  are 
L :  Low,  M :  Medium,  and  H :  High.  As  indicated  in  the  table,  both  NTI  and  MCC 
have  the  most  of  the  High  marks. 

Table  4-3.  Test  Summary 


■KiiHi 

VisionShaoe _ _ 

Genoal 

1)  flexibility  to  for  integration  into  CAD  A  system; 

iM 

M 

M 

M 

2)  speed; 

;m  , 

H 

L 

H 

3)  software  quality  and  stability 

H 

M 

M 

L 

4)  Application  Program  Interface  (API) 

M 

L 

L 

L 

Preprocessing  Test 

1)  box  finding  capability; 

L 

H 

H 

M 

2)  text  finding  capability; 

H 

H 

H 

M 

3)  key  field  correlation 

L 

H 

H 

M 

4)  line  and  box  removal; 

H 

H 

H 

H 

Character  Recognition  Test 

1)  character  segmentation  accuracy; 

M 

H 

M 

L 

2)  character  recognition  accuracy; 

M 

H 

H 

L 

3)  neural  network  trainability  for  new 

character  styles 

not  tested 

not  tested 

not  tested 

not  tested 

Postprocessing  Test  Criteria 

1)  heuristic  capability 

not  tested 

not  tested 

not  tested 

not  tested 

2)  validation  capability 

not  tested 

not  tested 

not  tested 

not  tested 

Key:  H  :  High  M  :  Medium  L  :  Low 
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SECTION  5 
CONCLUSIONS 


5.1  IMAGE  TESTING 

Several  important  conclusions  can  be  made  from  the  image  testing  conducted  in  this  phase  of  the 

project  The  following  paragraphs  describe  those  conclusions. 

1 .  The  CADA  tools  for  field  testing  should  allow  the  operators  to  configure  the 
techniques  and  parameters,  due  to  differences  in  data  characteristics  and  quality 
assurance  standards  across  various  organizations. 

2.  A  combination  of  five  techniques  was  identified  for  a  high  level  of  quality  control 
standards  that  should  be  applicable  to  new  data  received  from  the  contractors. 

Using  this  combination  on  the  limited  suite  of  test  data,  a  false  accept  (FA)  rate  of 
less  than  1  %  and  a  false  reject  rate  of  22%  was  obtained. 

3.  Objective  criteria  must  be  used  in  the  QA  process  for  CALS  data.  The  automated 
quality  analysis  techniques  can  be  most  effectively  used  if  the  quality  control  is 
based  on  objective  criteria  like  conformance  to  ANSI  standards  for  line  widths, 
absence  of  noise,  lack  of  fading,  etc.,  rather  than  on  subjective  judgement  and 
interpretation  of  the  contents  of  data. 

5.2  IDENTIFICATION  TESTING 

The  following  conclusions  can  be  made  for  the  identification  recognition  testing. 

1.  The  application  of  the  neural  network  techniques  do  hold  promise  for  automated  ID 
recognition,  and  three  vendors’  software  have  been  identified  that  give  good  results 
recognizing  hand  printed  and  machine  printed  information. 

2.  The  technology  is  mature  enough  to  automate  the  identification  recognition 
processing  for  the  drawings  that  meet  the  ANSI  standards. 

3.  The  type  of  characters  trained  by  each  individual  neural  network  affects  the 
recognition  results.  Different  types  of  characters  include  machine  printed, 
constrained  hand-printed  and  unconstrained  hand-scripted.  Neural  networks 
perform  well  if  the  target  characters  can  be  classified  as  one  of  these  types. 

4.  The  neural  network  provided  by  NTI  is  much  less  sensitive  about  the  type  of 
characters  to  be  recognized,  and  the  recognition  accuracy  of  the  NTI  engine  is  better 
than  those  of  other  companies. 
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SECTION  6 

RECOMMENDATIONS 


6.1  IMAGE  TESTING 

1 .  It  is  recommended  that  all  new  data  prepared  by  government  contractors  should  be 
required  to  adhere  to  the  government  recommended  ANSI  Y14.2M  standard.  This 
standard  defines  the  line  conventions  and  lettering  requirements  for  engineering 
drawings.  If  all  contractors  are  required  to  adhere  to  this  standard  by  contract,  the 
use  of  image  quality  techniques  will  produce  the  best  results. 

2 .  It  is  recommended  that  the  quality  requirements  of  new  data  procured  by  the 
government  contractors  be  clearly  specified  in  the  contract  in  objective  terms. 
Currently,  the  quality  assurance  is  based  on  subjective  criteria  like  “legibility  and 
reproducibility,”  which  are  not  well  suited  for  automated  quality  assurance 
procedures.  These  procedures  would  be  most  effective  if  the  contract  specifies 
objective  criteria  like  acceptable  limits  of  some  of  the  noise  parameters  used  in  this 
test. 

6.2  IDENTIFICATION  TESTING 

The  following  recommendations  are  made  for  the  future  improvements  on  the  identification 
recognition. 

1 .  It  is  recommended  that  more  effort  be  devoted  to  the  pre-processing  area.  There 
are  no  readily  available  COTS  software  packages  that  can  be  directly  used.  The 
CADA  task  should  develop  the  needed  pre-processing  capability.  Locating  the 
revision  number  in  an  engineering  drawing  is  by  fiir  the  most  difficult  task  of  all. 
Since  the  revision  number  column  holds  all  revision  history,  it  is  dynamically 
expanded  each  time  the  revision  is  made  to  the  drawing.  The  method  to  locate  the 
revision  number  correctly  has  yet  to  be  defined. 

2.  It  is  recommended  that  additional  Postprocessing  of  the  data  be  performed  during 
the  integration  phase  of  the  CADA  task. 

3.  It  is  recommended  that  contracts  for  procurement  of  new  engineering  drawing  data 
specify  that  ANSI  standards  Y  14.1  and  Y  14.2  be  met.  This  especially  applies  to 
the  format,  letter,  and  line  requirements  of  the  ANSI  standards. 
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SECTION  7 

ACRONYMS  AND  GLOSSARY 


ANSI 

API 

CAD 

CADA 

CALS 

CCITT 

CDRL 

CIT 

CR 

CTC 

CTN 

CTNO 

DID 

DoD 

DSREDS 

EDGARS 

EDMICS 

EDP 

FA 

FA 

FF 

GB 

ICR 

ID 

ICR 

LAN 

OCR 

PM 

ppm 

QA 

QC 

SOW 

TIFF 


American  National  Standards  Institute 

Application  Program  Interface 

Computer-Aided  Design 

Computer-Assisted  Data  Acceptance 

Computer-aided  Acquisition  and  Logistic  Support 

Consultative  Committee  for  International  Telephony  and  Telegraphy 

Contract  Data  Requirements  List 

Consumable  Item  Transfer 

Compression  Ratio 

CALS  Technology  Center 

CALS  Test  Network 

CALS  Test  Network  Office 

Data  Item  Description 

Department  Of  Defense 

Digital  Storage  and  Retrieval  Engineering  Data  System 

Engineering  Data  Computer  Assisted  Retrieval  System 

Engineering  Data  Management  Information  Control  System 

Engineering  Drawing  Package 

False  Accept 

False  Reject 

File  Format 

Gigabyte 

Intelligent  Character  Recognition 

Identification  Data 

Intelligent  Character  Recognition 

Local  Area  Network 

Optical  Character  Recognition 

Program  Management/Project  Manager 

parts  per  million 

Quality  Assurance 

Quality  Control 

Statement  of  Work 

Tagged  Image  File  Format 
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SECTION  8 

REFERENCE  DOCUMENTS 


Technical  Report  -  Testing  Techniques  for  Data  Acceptance  Procedures.  July  12, 1991. 

CCITT  Blue  Book,  Vol  VII  -  Fascicle  V1I.3,  Terminal  Equipment  and  Protocols  for  Telematic 
Services,  Recotnm.  T.0-T.63. 

ANSI  Y14.1-1975,  American  National  Standard  for  Engineering  Drawings  and  Related 
Documentation  Practices;  American  Society  of  Mechanical  Engineering 

ANSI  Y14.2M-1979,  American  National  Standard  for  Engineering  Drawings  and  Related 
Documentation  Practices;  American  Society  of  Mechanical  Engineering 

MIL-M-9689D  Military  Specification  Microfdrning  of  Engineering  Documents,  35  MM, 
Requirements  for 

MIL-STD-1840A,  Military  Standard,  Automated  Interchange  of  Technical  Information,  20, 
December  1988 

MIL-R-28002A  Raster  Graphics  Representation  in  Binary  Format,  Requirement  for,  30, 
November  1990 
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APPENDIX  A 
TEST  PLAN 


1.  Introduction  and  Background 

Computer-assisted  Data  Acceptance  (CADA)  procedures  for  Government  acceptance  of  CALS 
Raster,  Type  I  data  have  been  prepared.  Implementation  of  these  procedures,  in  the  acceptance  of 
CALS  data,  requires  that  image  and  identification  recognition  techniques  be  thoroughly  analyzed, 
tested  and  integrated  prior  to  conducting  field  operational  testing  at  the  Services  repositories. 

2.  Objective 

This  plan  defines  the  procedures  required  to  conduct  testing  of  image  quality  algorithms  and 
candidate  vendor  Intelligent  Character  Recognition  (ICR)  techniques  and  engines.  A  report  will  be 
prepared  which  shows  the  results  of  testing  both  existing  DSREDS  and  EDGARS  data  and  CALS- 
equivalent  quality  data. 

3.  Standards  and  Spccincations 

3. 1  MIL-STD-1840A,  Military  Standard,  Automated  Interchange  of  Technical  Information,  20, 
December  1988 

3.2  MIL-R-28002A  Raster  Graphics  Representation  in  Binary  Format,  Requirement  for,  30, 
November  1990 

4.  Procedures 

The  scope  of  the  testing  effort  requires  that  input  data  for  image  analysis  and  identification 
recognition  tests  be  prepared  or  obtained  and  used  as  the  basis  for  evaluating  image  quality 
algorithms  and  candidate  vendors’  ICR  techniques  and  products.  The  procedures  include  image 
quality  and  identification  recognition  testing. 

4.1  Image  Quality  Algorithms  Test  Procedures 

a.  Obtain  representative  test  data  from  DSREDS  and  EDGARS. 

b.  Create  or  obtain  CALS  eciuivalcnt  quality  test  data. 

c.  Classify  the  test  data  prior  to  testing. 

d.  Select  image  algorithms  for  evaluation  and  te.sting. 
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e.  Conduct  individual  and  integrated  testing  of  the  algorithms  on  DSREDS/EDCARS 
data  and  on  high  quality  CALS-compliant  data. 

f.  Document  and  analyze  the  test  results. 

g.  Prepare  results  and  analysis  for  delivery  in  a  test  report. 

4.2  Identification  Recognition  Test  Procedures 

a.  Prepare  a  suite  of  test  data  from  the  representative  DSREDS  and  EDGARS  samples. 

b.  Prepare  a  suite  of  test  data  to  represent  CALS-compliant  data. 

c.  Identify  ICR  vendors  for  product  evaluation. 

d.  Prepare  specification  for  identification  recognition  evaluation  testing. 

e.  Conduct  testing  of  identified  vendor  products  at  the  Army  CTN  Test  Bed  on  both 
existing  DSREDS  /  EDGARS  data  and  CALS  equivalent  quality. 

f.  Document  and  analyze  the  test  results. 

g.  Prepare  results  and  analysis  for  delivery  in  a  test  report. 

5.  Facilities  and  Equipment 

The  following  facilities  and  equipment  will  be  used  when  this  test  plan  is  executed: 

5.1  Facilities 

The  Army  CTN  Test  Bed  located  at  the  PM  JCALS  CALS  Technology  Center  (CTC),  Ft. 
Monmouth,  NJ. 

5.2  Hardware: 

a.  Sun  Sparc  workstation  370  computer  system  with  9-track  tape  drive; 

b.  ComputerVision  CDS  3000  with  9- track  tape  drive; 

c.  Apple  Macintosh  Ilex  computer  system; 

d.  Gateway  PC  486  computer  system; 

e.  Intergraph  InterPro  2000  computer  system;  and 
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f .  Ethernet  local  area  network. 

5.3  Software: 

a.  Application  Software  routines  in  "C"  to  develop  and  integrate  image  analysis  routines 

wiA; 

b.  User  interface  routines; 

c.  CCm  Decompression  routines; 

d.  CTN  Tape  tool  routines; 

e.  Intergraph  conversion  routines;  and 

f.  Candidate  vendor  identification  recognition  routines. 

6.  Deliverables 

A  Techniques  Test  Report  entitled  Phase  III:  Computer-Assisted  Data  Acceptance  will  be 
delivered  on  26  May  1992.. 

7.  Schedule 
See  Figure  A-1. 
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APPENDIX  B 

IMAGE  ANALYSIS  RESULTS 


This  appendix  gives  detailed  results  for  the  following  two  cases  of  image  quality  testing: 

1.  Image  Data  Set  1  using  “High  Quality”  criteria. 

2.  Image  Date  Set  2  using  “Low  Quality”  criteria. 
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APPENDIX  C 

IDENTIFICATION  RECOGNITION  RESULTS 


This  appendix  gives  detailed  results  for  the  following  two  cases  of  identification  recognition  testing 
for  the  vendors  tested. 

1 .  K)  Date  Set  2  existing  DSREDS/EDCARS  Data: 

•  MCC  results, 

•  Nestor  results, 

•  NTI  results,  and 

•  Visions  hape  results. 

1 .  ID  Date  Set  3  “High  Quality”  Data: 

•  MCC  results, 

•  Nestor  results, 

•  NTI  results,  and 

•  VisionShape  results. 
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MCC  Data  Set  2  Test  Results 


MCC  Data  Set  2  Test  Results 


Nestor  Data  Set  2  Test  Results 


Nestor  Data  Set  2  Test  Results 


NT!  Data  Set  2  Test  Results 


NTI  Data  Set  2  Test  Results 
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Nestor  Data  Set  3  Test  Results 


1 

II 

1 

II 

o 

1 

II 

CO 

00 

CO 

1 

II 

1 

1 

1 

10E1 

1 

E 

0 

6OE10 

iZ 

O 

r— • 

LL 

O 

CM 

U. 

o 

CM 

u. 

O 

1 

CM 

Li- 

0 

CM 

o 

r“ 

LL 

O 

CO 

IL 

o 

CM 

LL 

O 

1 

00 

o 

o 

O 

484 

1474 

CO 

CO 

•r- 

57% 

s 

CM 

1 

CM 

CO 

CO 

_j 

0. 

CO 

CM 

ri. 

) 

CO 

136-27-265 

Cl  36-29-1 

BLL-4-  J4 

LL-4L-LLL15L 

7550289 

7551818 

7551830 

P9234710 

%  Char.  Correct 

Total  Fields 

%  Field  Correct 

IPL  136-27-261 

136-27-262 

136-27-265 

Cl  36-29-1 

Cl  36-41 -194 

136-41-1951 

PL-1 36-41 -1968 

CO 

CO 

o 

■t— 

1 

O) 

in 

7550289 

7551818 

7551830 

o 

'r— 

CO 

CM 

CD 

CL 

o 

1 

CO 

in 

o 

o 

o 

8Z361 

81361 

81361 

s 

CO 

00 

00 

CO 

yr- 

CO 

81361 

o 

CM 

CD 

O 

CM 

CD 

o 

CM 

CD 

1 

Jd 

CO 

T— 

00 

CD 

CO 

00 

Jd 

CO 

T— 

00 

CO 

CO 

00 

S 

CO 

CO 

T“ 

00 

CO 

CO 

•r— 

00 

CO 

00 

O 

CM 

CD 

O 

CM 

CD 

rSi 

o 

O 

o 

o 

o 

o 

o 

1 

1 

1 

1 

1 

1 

1 

OQ 

lU 

CQ 

CM 

LL 

1 

1 

ffi 

o 

O 

CD 

CD 

Q 

1 

LL 

1 

1 

CO 

ir> 

o 

X3 

CO 

CO 

If) 

o 

•Q 

|d015r267 

CM 

00 

LO 

O 

TO 

00 

a> 

in 

o 

X3 

|d015r299 

r^ 

CM 

C 

in 

■ 

o 

T3 

o 

CM 

in 

o 

TO 

|d015r487 

o 

CM 

m 

o 

TO 

d015r540 

? 

LO 

1. 

I 

1 

1 

1 

1 

1 

1 

1 

C-18 


NTI  Data  Set  3  Test  Results 


LJ009H029  b _ E _ 0  19200 _ 19200  0  12916786 _ 12916786 

D009R032  C _ C _ 0 _ 19200 _ 19200 _ 0 _ 12916786 _ 12916786 

D009R100  B _ B _ 0  19205 _ 19205 _ 0  5009274  5009274 

D009R102  B _ B _ 0 _ 19205 _ 19205  0 _ 501 3565 _ 5013565 

D009R104  IB  Ib  I  o|  19205|  19205  _ 0 _ 5013611  5013611 


U-  LL. 

o  o' 

<D  T- 


O)  1- 

UL  LL 

O  O 

___CD  _  -r- 

o  o  o  o 


^  O  Tf  r-* 
C\J  00  C\J  CD 

Cp  ^  op  CM 

T  CO  T 
CO  CM  CO  ^ 
CM  ^  CM  CD, 

5  Q  5  $2 


NTI  Data  Set  3  Test  Results 


c 

11 

1 

If 

1 

1 

1 

1 

1 

1 

1 

1 

I 

1 

1 

1 

I 

UL 

O 

1 

CM 

Li. 

o 

CM 

6  OF  10 

T- 

LL 

o 

CM 

LL 

O 

LL 

O 

CM 

LL 

o 

CM 

LL 

O 

1 

1 

1 

60F  10 

O 

i 

1 

o 

o 

1 

1 

1 

1 

1 

PL1 36-27-261 

ptjl 

!l 

n 

|| 

PL136-41-1968 

5-19-11083 

7550289 

7551818 

7551830 

P923471 0 

69 

1474 

in 

a> 

y— 

CO 

311 

90% 

PL136-27-261 

1 

11 

U136-41-194 

136-41-1951 

PL1 36-41 -1968 

5-19-11083 

7550289 

7551818 

7551830 

P9234710 

Total  Char.  Errors 

Total  Characters 

%  Char.  Correct 

Total  Field  Errors 

■D 

<D 

Ll 

15 

b 

h- 

%  Fields  Correct 

o 

o 

o 

J 

I 

II 

o 

o 

1 

1 

1 

1 

1 

81361 

•r- 

CO 

CO 

oiabi 

81361 

81361 

81361 

19204 

"M- 

O 

CM 

CJ> 

T— 

o 

CM 

Oi 

1 

1 

r 

r 

81361 

CO  < 

5  c 

D  Jo 
O  CO 

D  S 

<D 

CO 

s 

81361 

19204 

o 

CM 

O) 

1 

1 

1 

i 

1 

1 

1 

1 

1 

1 

1 

1 

1 

n 

1 

1 

CD 

LU 

GO 

o  c 

II 

1 

Q 

1 

CM 

CC 

in 

o 

a 

s 

CM 

r 

n 

f— 

o 

CD 

CM 

X 

LO 

r— 

o 

DM 

CM  a 

X  a 

LO  ir 

•“  T- 

o  c 
D  C 

C)^D15R299 

r^ 

CO 

X 

in 

T— 

o 

D 

o 

X 

n 

r— 

O 

DJ 

? 
X  1 
n 

5 

Dj 

o 

CM 

n 

X 

n 

r- 

o 

D 

o 

S 

X 

n 

5 

Dj 

1 

1 

1 

1 

1 

1 

1 

C-21 


VislonShape  Data  Set  3  Test  Results 


6 


Total  Field _ 91^ 

%  Fields  Correct  37% 


Test  Report:  Phase  III  Computer-Assisted  Data  Acceptance 

APPENDIX  D 

CADA  IDENTIFICATION  RECOGNITION  TEST  REQUIREMENTS 

SPECIFICATION 


1  CHARACTER  STRING  IDENTIFICATION  PROCESSING  DESCRIPTION 

Character  String  Identification  Processing  is  a  subtask  of  the  Computer  Aided  Data  Acceptance 
(CADA)  task  for  the  CALS  Technology  Center.  Character  String  Identification  Processing 
identifies  each  character  string  (regardless  of  fonts,  block  hand-printed  scripts;  character  size  is 
from  20x20  pixels  to  200x200  pixels)  in  an  engineering  drawing,  with  possible  background 
noises,  such  as  lines  and  boxes,  in  the  input  image  file.  The  input  is  an  image  file  in  TIFF  format. 
The  output  is  a  list  of  text  strings  as  it  was  originally  grouped.  The  text  string  may  contain  mixed 
alphabetical  characters  as  well  as  numerical  characters.  Characters  to  be  recognized  will  consist  of 
at  least  all  upper  case  alpha,  numeric  letters,  slash  sign  left  and  right  parenthesis  "(",  ")"  and 
dash  sign  There  will  be  no  exact  fonns  to  be  defined,  even  through  the  Title  area  of 
engineering  drawings  may  look  like  a  form. 

A  software  callable  library  and  an  API  (Application  Program  Interface)  will  be  delivered  for  CADA 
acceptance  testing.  The  system  wilt  also  include  a  Vendor  Training  Utility  Tool .  The  Vendor  will 
support  the  installation  of  the  software  package  and  provide  support  in  using  the  Training  Tools 
sufficient  to  enable  the  performance  of  the  evaluation  test  through  Marchl,  1992. 

The  API  will  include  all  functions  necessary  to  process  CADA  ID  recognition.  The  CADA  ID 
recognition  function  is  described  in  Section  3.0,  Processing.  A  preliminary  set  of  API  will  be 
provided  by  the  Vendor  and  approved  no  later  than  one  week  after  acceptance  of  the  purchase 
order.  The  functionality  described  within  the  attachment  supersede  API  in  the  case  of  any  conflicts 
between  the  two.  Any  changes  to  the  API  specification  shall  be  approved.  A  final  API 
specification  will  be  delivered  with  the  software  package. 

2  INPUT 

Input  to  the  character  string  processing  is  an  image  file  in  TIFF  format.  The  text  layout  is  in  a  free 
format.  The  image  may  consist  of  single  characters,  text  strings,  lines  and  boxes,  and  noises  when 
it  is  digitized.  An  example  of  the  Title  area  image  is  illustrated  in  Figure  1. 

The  government  will  provide  a  suite  of  engineering  drawing  data  that  the  Vendor  shall  utilize  for 
the  purpose  of  training  of  neural  networks. 
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3  TEXT  STRING  RECOGNITION  PROCESSING 

The  functional  processing  capability  of  this  package  is  to  isolate  and  recognize  each  text  field  in  the 
image  provided.  The  image  to  be  analyzed  may  contain  single  characters,  text  strings,  lines  and 
boxes,  and  some  noises  when  it  was  digitized.  Text  strings  may  contain  machine-printed,  as  well 
as  hand-printed  letters  of  different  sizes  and  fonts.  A  character  is  recognized  if  the  recognition 
confidence  level  computed  by  the  recognition  engine  has  passed  the  input  threshold  value. 

If  there  are  boxes  in  the  image  file,  the  text  strings  recognized  are  grouped  and  reported  by  box. 
Text  is  arranged  from  left  to  right  and  top  to  bottom  order  in  the  output  list.  A  vertical  bar  "I"  is 
used  as  a  delimiter,  separating  text  strings  for  the  different  boxes.  The  associated  confidence  level 
of  each  character  recognized  will  be  provided. 

3.1  Second  Choice  of  recognition  for  Each  Character 

For  each  character  to  be  recognized,  there  is  a  second  choice  for  the  recognition.  This  process  is 
very  similar  to  the  processing  defined  in  the  above  section,  except  that  the  characters  with  lower 
recognition  confidence  level  will  be  put  in  the  output.  The  calculated  confidence  level  for  each 
second  choice  is  reported  as  well. 


3.2  Pre-processing  (PP) 

This  process  is  to  identify  all  text  string  areas  separated  by  a  structure.  The  coordinates  of  the  text 
image  area  is  identified  by  the  preprocessing  function  so  that  the  text  strings  can  be  extracted  and 
send  to  a  character  recognition  engine. 


4  OUTPUT 

The  output  of  the  package  is  described  in  two  parts:  hard  copy  print  out  and  application  program 
interface. 

4.1  Character  Recognition  Hard  Copy  Print  Out 

The  output  for  the  recognized  characters  consists  two  major  parts:  a  list  of  characters  of  primary 
choice  and  a  list  of  characters  of  secondary  choice.  Each  list  is  a  string  of  recognized  text  Each 
list  is  also  associated  with  an  array  of  confidence  level  percentage  values  for  each  character.  Each 
list  of  the  recognized  text  strings  is  arranged  from  left  to  right  and  top  to  bottom  order  in  the  output 
list.  For  example,  if  the  input  file  is  as  defined  in  figure  1 ,  the  output  string  for  the  first  choice 
recognized  by  a  good  system  will  be: 

PRODUCT  SPECinCATION  MICROCIRCUITS,  DIGITALI3-73ISIZE  AIFSCM  NO.  NO. 
25500DWG  NO.  3100109-007ISCALE  NONEISHEET  1  OF  79 
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An  array  of  confidence  level  percentage  value  for  each  character  is  arranged  in  the  same  sequence 
as  the  characters  in  the  text  list.  The  corresponding  output  array  for  the  recognition  confidence 
level  of  the  above  text  string  has  a  length  of  101  entries  (one  confidence  level  entry  for  each 
character  recognized). 

4.2  Preprocessing 

The  output  of  the  preprocessing  is  a  list  of  X,Y  coordinates  of  the  upper-left  and  lower-right 
comers  of  the  identified  rectangular  boxes  as  well  as  the  text  images  areas  within  each  rectangular 
box. 

4.3  Application  Program  Interface  (API) 

The  title  block  processing  is  invoked  by  calling  the  routine  ID_processing.  Calling  this  function 
will  initiate  processing  of  a  title  block.  This  function  will  return  an  integer  value.  A  return  value  of 
zero  will  indicate  no  error,  non-zero  returns  will  indicate  an  error.  A  table  of  possible  error  return 
values  and  their  associated  meanings  will  be  supplied  to  Accurate  at  software  delivery.  The  routine 
provided  will  use  the  following  API: 

int  ID_process  (data_file_name,  block_data) 
char  *data_file_name; 

TTTLEBLOCK.DATA  *block_data; 

The  input  to  ID_Process  function  will  consist  of: 

data_file_name:  A  pointer  to  a  string  (null  terminated)  containing  the  name  of  the  TIFF  file  that 
contains  the  title  block  image.  If  the  file  is  not  in  the  current  working  directory,  the  full  pathname 
should  be  provided. 

The  outputs  of  the  ID_Process  function  will  consist  of: 

block_data:  A  pointer  to  a  data  structure  that  will  contain  the  recognition  results  of  the  title  block 
processing.  The  data  interested  in  a  title  block  consists  of  rectangular  boxes,  the  texts  in  each  box, 
and  text  strings  not  in  any  box.  The  block_data  structure  is  a  "C"  data  structure  defined  as  follows: 

typedef  struct  title_block_data 

{ 

intn_box_fields;  /*  number  of  boxes  returned  for  this  block  */ 
intn_box_fields_allocated; 

BOX_FIELD*box_fields;  /*  points  to  box-field  */ 

intn_text_fields;  /*  number  of  fields  returned  for  this  block 
not  in  any  box  */ 
intn_text_fields_allocated; 
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TEXT_FIELD*text_fielcis; 

}TITLE_BLOCK_DATA 

typedef  struct  box_fields 

inttlx,  tly,  brx,  bry;  /*  location  of  the  box  */ 

intn_text_fields;  /*  number  of  text  fields  returned  for  this  box  */ 

int_text_fields_allocated; 

TEXT_FIELD*text_fields; 

BOX_FIELDS 

the  TEXT_FIELDS  type  is  defined  as  follows: 
typedef  struct  text_field 

{ 

inttlx,  tly,  brx,  bry;/*  location  of  text  field  in  TIFF  image  *^ntr_char_results;/*  number  of  chars 

in  this  text 

field*/ 

intn-chars_results_allocated; 

CHAR_RESULTS  *char_results; 

}TEXT_FIELD 


The  CHAR_RESULTS  type  is  defined  as  follows: 
typedef  struct  char_results 

inttlx, tly ,brx,bry;/*  location  of  character  bounding  box 
with  respect  to  text_field  */ 

charchar_class[2];/*  first  two  choices  of  recognized  class 
of  character  */ 

intconfidence[2];/*  confidence  of  each  choice  */ 

}CHAR_RESULTS 

5  PREACCEPTANCE  AND  ACCEPTANCE  TEST 
5.1  Preacceptance  Test 

Preacceptance  tests  are  performed  by  the  Vendor  at  their  facility  before  the  system  to  be  evaluated 
is  delivered.  The  Vendor  shall,  as  a  minimum,  demonstrate  the  system  by  recognizing  all  text 
strings  for  five  input  files  of  the  Title  area  cropped  from  the  entire  engineering  drawing  image. 
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5.2  Acceptance  Test 

Acceptance  testing  was  performed  at  the  end  of  the  evaluation  period,  on  or  before  March  1, 1992. 
The  input  to  the  acceptance  test  utilizes  Title  area  images  of  a  similar  format  layout  to  that  used  in 
the  Preacceptance  test,  and  with  the  equivalent  image  quality.  The  number  of  such  images  for  the 
acceptance  test  is  estimated  around  5000.  The  accuracy  of  the  delivered  system  is  at  least  at  90% 
level  for  the  primaty  choice. 
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Figure  D-1.  Sample  Input  Image 
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